Search and add queries that use Lucene specific optimizations about search-benchmark-game

[fulmicoton]

Right... So Lucene introduced two step queries a long time ago. The idea is that scorer can express that they have some notion of .advanceWithFalsePositives() that is cheap and a checkIfReallyInDocset() function that is expensive to compute.

This is typically the case for a phrase query. .advanceWithFalsePositives() is typically just an intersection of the terms. checkIfReallyInDocset() requires to decode positionos and check if the terms align correctly somewhere in the document.

Now something interesting happens when running the query "a b" AND c.
The two step query system makes it possible to only decode positions for document that contain C.

If you scroll down in the benchmark that's precisely why added the query +"the who" +uk which is the hero scenario for this (because the and who are frequently in the same document.).
Lucene is indeed outperforming tantivy on that kind of query.

It would be interesting to find other kinds of examples

[fulmicoton]

added a query that benefits heavily from the two step queries.

[usamec]

@fulmicoton any deeper description about what is exactly going on here? (link to some lucene documentation is enough).

[usamec]

Thanks, and is tantivy planning to add this?

[fulmicoton]

Yes at one point. But right now, I don't think the extra complexity is worth the performance boost on specific queries. If a performance critical user shows up, I'll be happy to reproritize.

It's however a good idea to list these corner case where Lucene is faster than tantivy and show them in the bench.