SQLFetchScroll() calls are progressively slower with "ORDER BY" query when the SQL Server table has a primary key defined. And with a table without primary key, SQLFetchScroll() are performing consistently and faster. A query without an "ORDER BY" is showing the same behavior (progressively slower ) regardless whether a primary key is defined on the table or not. 

Some examples of these behavior found from the trace logs are given below. Check the "Time elapsed" part:

SQLFetchScroll() is progressively slower when fetching result set against table WITH primary key for SELECT query with "ORDER BY"-
2019-06-12 07:52:33.988547 (Time elapsed in microseconds :    88884) ppid=20485:pid=21583:tid=140417771779     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 07:52:34.105283 (Time elapsed in microseconds :   111976) ppid=20485:pid=21583:tid=140417771779     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 07:52:34.244798 (Time elapsed in microseconds :   134710) ppid=20485:pid=21583:tid=140417771779     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 07:52:34.407466 (Time elapsed in microseconds :   157723) ppid=20485:pid=21583:tid=140417771779     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)

SQLFetchScroll() is consistent and fast  when fetching result set against table with NO primary key for SELECT query with "ORDER BY":
2019-06-12 08:02:20.199863 (Time elapsed in microseconds :    22981) ppid=22881:pid=23874:tid=140298961196     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:02:20.227517 (Time elapsed in microseconds :    22743) ppid=22881:pid=23874:tid=140298961196     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:02:20.254758 (Time elapsed in microseconds :    22432) ppid=22881:pid=23874:tid=140298961196     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:02:20.281789 (Time elapsed in microseconds :    22186) ppid=22881:pid=23874:tid=140298961196     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)

SQLFetchScroll() is progressively slower when fetching result set against table WITH primary key for SELECT query without  "ORDER BY":
2019-06-12 08:09:33.266405 (Time elapsed in microseconds :    89521) ppid=22881:pid=25116:tid=140616344667     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:09:33.383557 (Time elapsed in microseconds :   112228) ppid=22881:pid=25116:tid=140616344667     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:09:33.523991 (Time elapsed in microseconds :   135570) ppid=22881:pid=25116:tid=140616344667     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:09:33.686338 (Time elapsed in microseconds :   157383) ppid=22881:pid=25116:tid=140616344667     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)

SQLFetchScroll() is progressively slower when fetching result set against table with NO primary key for SELECT query without "ORDER BY":
2019-06-12 08:12:06.894300 (Time elapsed in microseconds :    89139) ppid=22881:pid=25538:tid=139859270121     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:12:07.015621 (Time elapsed in microseconds :   116388) ppid=22881:pid=25538:tid=139859270121     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:12:07.164752 (Time elapsed in microseconds :   144130) ppid=22881:pid=25538:tid=139859270121     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)
2019-06-12 08:12:07.341358 (Time elapsed in microseconds :   171755) ppid=22881:pid=25538:tid=139859270121     EXIT  SQLFetchScroll  with return code 0 (SQL_SUCCESS)