Tree-sitter highlighting no longer takes query range/specificity into account about neovim

[clason]

• "Mom, can we get more treesitter tests?"
• "We have treesitter tests at home".

treesitter tests at home

[lewis6991]

This is most definitely because we've switched from iter_captures to iter_matches.

I don't think this is intended. We can add the repro steps as a testcase to prevent this happening again, but I'm surprised our current tests don't capture this in some way.

[lewis6991]

• "Mom, can we get more treesitter tests?"
• "We have treesitter tests at home".

treesitter tests at home

In that case, would it be worth adding nvim-treesitters CI to Neovim's CI? It would certainly boost coverage.

[gpanders]

How did this even work before with iter_captures? Does ts_query_cursor_next_capture order iterated nodes based on "specificity"? As far as I can tell, we are not doing anything on the Neovim side to do this ordering. The old highlighter that use iter_captures just applied extmarks in the exact order returned from the iterator.

[gpanders]

It'd also be fine to just not consider specificity anymore, that's totally fine but some queries need to be reworked then (e.g. some of C's highlights) and this should be documented to avoid further confusion

I do think we should do this too and make the order defined in the query the only thing that influences priority. Having two separate metrics that influence priority is confusing. But in the interest of supporting BWC we should still fix this in the meantime.

[clason]

In that case, would it be worth adding nvim-treesitters CI to Neovim's CI? It would certainly boost coverage.

Actually, I'd claim we should do the converse: add a treesitter highlight test suite (mirroring tree-sitter test, but using our highlighter instead of the Rust crate -- either tree-sitter's or theHamsta's) exercising the bundled queries and expose it in a way that nvim-treesitter can steal leverage.

[lucario387]

How did this even work before with iter_captures

iter_captures return the results based on the order of a node in the tree, so the outer appears before the inner and so on

[lewis6991]

The repro in the OP actually caused the highlighter to error when I added it as a testcase. Haven't got time to debug, but using iter_captures as before works fine.