tokee / lucene Goto Github PK

For 10 million terms, the startup time is about 20 minutes on a fast machine. The primary cause is non-sequential term lookup and cache misses. Increasing the cache to hold all terms in memory works fine, but takes just as much memory as a standard Lucene sort.

Solution:

• Process the array with term ordinals in cache-sized chunks
• Sort each chunk with merge sort, which results in sequential lookup of the terms as there are no cache misses
• Perform a merge of all the chunks by using a heap to determine the next term ordinal

Provided the number of chunks is lower than the cache-size, the end result is perfect cache-usage and perfect sequential access for the merge sort step. The term access for the heap-based merge at the end will depend on whether the locale sort is well aligned to unicode sort. This should work fairly well for a number of languages and probably not well at all for some languages. Experiments will show how this turns out.

The unit-test for Exposed aka LUCENE-2335 is now capable of testing sorting for indexes with deleted documents. The unit-test fails if there are deletions, stating that the order differs for Exposed vs. the default Lucene sorter. The cause has yet to be determined.

While experimenting on an in-house index with ~5M terms in the sort field, the mapping from docIDs to Term seems wrong. The culprit is of course the docID->termOrder mapper.