Use SOLR cursors for pagination of large set of results about ads

[vsudilov]

Hi Alberto,

Seems like a straightforward improvement.

On Friday, July 22, 2016, Alberto Accomazzi [email protected]
wrote:

The search engine behind the ADS API is SOLR, which has performance
problem with queries that require "deep pagination," i.e. the retrieval of
records way down the list of results. The issues related to the degraded
performance are explained here:
https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results

To improve the situation, SOLR implements the notion of cursors (also
explained in the page above) which mitigate the pagination problem for
queries generating a large set of results which need retrieving. I believe
an implementation of multi-page retrieval based on cursors would be more
robust for a few reasons:

1. better performance: avoids recomputing long result list just to
   fetch items way down in the list (which could generate a timeout)
2. consistent results: if the index is modified during the follow-up
   queries, some results might be duplicated or skipped using the current
   approach, whereas this should not happen with the user of cursors

I tested this approach against our search engine and it seems to work but
does require sort to include the unique key id as a tie-breaker when
creating a list of result (e.g. sort=date asc,id asc).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#70, or mute the thread
https://github.com/notifications/unsubscribe-auth/ADV6ZAmDj3S3clLqw6a8F8fIXcvJXHVTks5qYSWvgaJpZM4JTGJR
.

[vsudilov]

Currently sort defaults to None, which according to https://github.com/adsabs/adsabs-dev-api/blob/master/search.md returns results sorted by relevancy.

Using cursorMark requires defining sort, which would change this default behavior. Is there a way to explicitly specify "relevancy" as a sort strategy?

I'm going to continue implementing support for cursorMark, but it seems like it only makes sense to use it if sort is not None.

Thoughts?

[aaccomazzi]

The default SOLR sort order is score desc so you could explicitly set this as primary if sort is None