Large query fails to return more than 2000 articles about ads

[vsudilov]

Hi, thanks for the report.

I suspect what's going on here is related to the fact that the ADS solr service re-writes rows if it is greater than a certain value as a safety measure: https://github.com/adsabs/solr-service/blob/master/solr/views.py#L59

When that happens, the ads client isn't aware of that re-write and expects its own rows to be definitive. I'm thinking the best way to handle this is to throw up a warning if the responseHeader doesn't contain the same data as self.query.

In general, setting rows to a reasonable value and iterating over the pages of results is the preferred way to query the database.

[barentsen]

@vsudilov I confirm that setting rows=2000 and max_pages=10 allows me to circumvent the bug, thanks.

In [1]: import ads
In [2]: qry = ads.SearchQuery(q='pub:"Monthly Notices of the Royal Astronomical Society" pubdate:"2015"', rows=2000, max_pages=10)
In [3]: articles = list(qry)
In [4]: len(articles)
Out[4]: 3140

[barentsen]

Thx++!