Implement syntax highlighting about elisp-tree-sitter

[ubolonton]

A simple implementation could be like below:

For initial highlighting:

• Run queries on the root node.
• Loop through all the captures, mapping capture-names to desired faces, calling add-text-properties.

For incremental changes:

• In a tree-sitter-after-change-functions hook, determine the affected ranges, using ts-changed-ranges .
• Run the queries again, over the affected ranges, using ts-set-byte-range.
• Loop through all the captures, calling add-text-properties again.

Further optimizations, like visible-only highlighting, or idle highlighting, can come later.

We don't even need to write the queries on our own (at least initially). Tree-sitter grammar repos now come with pre-defined queries, as they are aiming towards a new cross-editor highlighting standard (in which a common set of capture names is defined, e.g. @function, @function.name, @variable). Example: https://github.com/tree-sitter/tree-sitter-python/blob/master/queries/highlights.scm

[saskenuba]

It would be amazing to replace font-lock-mode fixed keywords highlight with the dynamic of the tree-sitter parser. I am willing to help after I finish my current project.

[ubolonton]

If things get hairy, we may look into font-lock itself and attempt to make it accept different "backends", or at least that is what I think in my ignorance.

EDIT: reading about font-lock from the link provided above it seems you can specify your own fontifying functions with font-lock-fontify-buffer-function which is called by font-lock-fontify-buffer. Then, it seems that getting syntax highlight work with current Emacs facilities should possible

font-lock (and jit-lock) pervasively assumes regular expressions are used for highlighting. Fitting tree-sitter directly into that is likely very hard. I think we should instead start with an alternative implementation based on the lower-level primitives around text properties.

is this issue planned to be resolved with Rust and just providing the bindings to Emacs lisp or are we meant to implement this using the current bindings available?

Is not obvious to me what functions one need to use in order to pass a query and receive the results.

There are 2 parts:

• Expose the new TSQuery functions through Rust. I'm starting to do this.
• Use these new functions to implement syntax highlighting (preferably by assigning text properties directly, instead of relying on the complex font-lock machinery).

[ubolonton]

The bindings are now available. The main functions are ts-make-query, ts-make-query-cursor, ts-query-matches, ts-query-captures.

There's also a convenient function for debugging (in tree-sitter-debug), which can be used like this:

;; Get names of functions/macros defined in a file.
(with-current-buffer "query.rs"
  (rust-mode)
  (tree-sitter-mode)
  (->> (tree-sitter-query [(function_item (identifier) @function)
                           (macro_definition (identifier) @macro)])
       (-map (lambda (capture)
               (pcase-let ((`[_ ,node] capture))
                 (ts-node-text node))))))

[codecoll]

FYI https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00801.html

[shackra]

More generally, I'm surprised, to say the least, that such a
significant job is being done without saying even a single word here,
let alone making sure the basic design is deemed reasonable by the
core developers. Do they really think this will make the acceptance
of the code easier?

Oof.

[shackra]

@codecoll thanks for bringing this to the project's attention

[ubolonton]

Syntax highlighting is documented here.

[ubolonton]

tree-sitter got a new interesting feature that should be used for this: tree-sitter/tree-sitter#444 (tree query).

[shackra]

basics of font-lock https://www.gnu.org/software/emacs/manual/html_node/elisp/Font-Lock-Basics.html#Font-Lock-Basics

[shackra]

How are we going to tackle this ticket? Should we first implement something that tries to work with font-lock?

If things get hairy, we may look into font-lock itself and attempt to make it accept different "backends", or at least that is what I think in my ignorance.

EDIT: reading about font-lock from the link provided above it seems you can specify your own fontifying functions with font-lock-fontify-buffer-function which is called by font-lock-fontify-buffer. Then, it seems that getting syntax highlight work with current Emacs facilities should possible

[shackra]

@ubolonton I'm little confused: is this issue planned to be resolved with Rust and just providing the bindings to Emacs lisp or are we meant to implement this using the current bindings available?

Is not obvious to me what functions one need to use in order to pass a query and receive the results.

In other words: what is the battle plan? My rust is nonexistent, so I cannot lend a hand writing bindings to leverage the TSQuery facilities from tree-sitter

[shackra]

Expose the new TSQuery functions through Rust. I'm starting to do this.

Excellent, any chance the queries can be passed as normal lisp lists? meaning this query:

(class_declaration
  name: (identifier) @the-class-name
  body: (class_body
    (method_definition
      name: (property_identifier) @the-method-name)))

is noted in Emacs Lisp as:

'(class_declaration
  :name  (identifier) @the-class-name
  :body (class_body
    (method_definition
      :name (property_identifier) @the-method-name)))

or something alike.

Use these new functions to implement syntax highlighting (preferably by assigning text properties directly, instead of relying on the complex font-lock machinery)

Regarding that, I was doing some tests, if you deactivate font-lock-mode evaluate the following code:

(defun my-fontify-region (beg end &optional loudly)
  (message (format "fontifying %s - %s (loudly %s)" beg end loudly)))

(setf font-lock-fontify-region-function 'my-fontify-region)

turning on font-lock-mode should give you the message:

Font-Lock mode enabled in current buffer
fontifying 1 - 323 (loudly nil)

so, we don't need to reinvent the wheel but neither rely on font-lock in a way that limit us, as I see it, we can implement our own syntax highlight functions for each grammar available and let font-lock use them instead of the defaults. Unless, of course, I'm missing something.

[ubolonton]

Excellent, any chance the queries can be passed as normal lisp lists?

Yes, eventually we would do that. In the initial version I'll just use the same syntax, to save time. Also, it's probably better to ask tree-sitter to support both name: and :name, instead of doing the conversion on our own.

Regarding that, I was doing some tests, if you deactivate font-lock-mode evaluate the following code:

turning on font-lock-mode should give you the message:

so, we don't need to reinvent the wheel but neither rely on font-lock in a way that limit us, as I see it, we can implement our own syntax highlight functions for each grammar available and let font-lock use them instead of the defaults. Unless, of course, I'm missing something.

That only covers how to highlight a region. I think the main complexity is in the change-handling mechanisms, i.e. which regions should be re-highlighted, when.

[shackra]

I see, I see.

Maybe I should deep dive on font-lock and see whenever these "edge cases" are already covered somehow while the bindings for TSQuery are on the works

[shackra]

from here is just a matter of start experimenting, thank you for your hard work. Let's see what I can come with.

[peterhoeg]

Apologies for necro-bumping, but I wasn't able to deduce from the syntax highlighting documentation if font-lock-mode is still used to do the actual high-lighting but tree-sitter then "feeds" it or if font-lock-mode is completely unnecessary with tree-sitter.