emacs-tree-sitter / elisp-tree-sitter Goto Github PK

View Code? Open in Web Editor NEW

803.0 26.0 71.0 1.36 MB

Emacs Lisp bindings for tree-sitter

Home Page: https://emacs-tree-sitter.github.io

License: MIT License

Shell 2.77% Rust 26.85% Emacs Lisp 68.11% PowerShell 1.95% Makefile 0.32%

rust emacs binding emacs-modules incremental parsing tree-sitter

elisp-tree-sitter's Issues

Args out of range error during tree-sitter-langs installation

For a couple of days now I've been getting the following error on a new TS install:

Warning (initialization): An error occurred while loading ‘/home/tim/.emacs.d/init.el’:

Args out of range: #<killed buffer>, 513, 16463596559

This is from Emacs 27.1, started with --debug-init
Config:

(unless (package-installed-p 'tree-sitter)
  (package-install 'tree-sitter)
  (package-install 'tree-sitter-langs))
(require 'tree-sitter)
(require 'tree-sitter-langs)

(First had it with use-package but went with a vanilla config to ensure it's not the setup)

Upcoming changes to tree-sitter query syntax

Hey there 👋

I’m filing this issue to make y’all aware that there are some upcoming changes to tree-sitter’s query syntax, stemming from tree-sitter/tree-sitter#615. The implementation of these changes should retain backwards compatibility, but going forward we’d recommend using the new syntax, as it’s (in our opinion!) more readable and powerful. We’ll be updating the highlighting.scm queries in the various tree-sitter language packages, so if you depend directly on them via submodules or the like, everything should Just Work.

If this is going to cause any grief for this project, please reach out on the tree-sitter issues board and let us know, and we’ll do our best to make the transition easy for you.

Thank you very much for all your work on this project: as an Emacs devotee, I personally am hugely excited about the prospect of using tree-sitter to power my Emacs!

cc @maxbrunsfeld

Syntax highlighting doesn't work for java annotations with scoped_identifier

To reproduce:

Create java class:

@Config.Source("any value")
public class Test {
  
}

Add the following to init.el:

(tree-sitter-hl-add-patterns 'java
    [
     ;; Scoped annotations
     (annotation
      (scoped_identifier) @parameter)

     (marker_annotation
      (scoped_identifier) @parameter)
     ])

Expected result:

Annotation highlighted using tree-sitter-hl-face:parameter

Actual result:

Config part highlighted using tree-sitter-hl-face:type face and .Source part highlighted using default face.

Emacs version:
GNU Emacs 27.1 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw3d scroll bars) of 2020-08-11

unexpected highlighting for python

tree-sitter-hl-mode cannot correctly hightlight class names starting with underscore.

minimal example:

class Foo:
    pass


class _Foo:
    pass

behavior:
class Foo has tree-sitter-hl-face:constructor face, but class _Foo has default face.

solution:
replace

((identifier) @constructor
 (#match? @constructor "^[A-Z]"))

in ./langs/queries/python/hightlight.scm with

((identifier) @constructor
 (#match? @constructor "^[A-Z_]"))

caveat:
I am not sure whether the current highlighting scheme is intentional (i.e., do not put tree-sitter-hl-face:constructor to private classes). If so, you can close the issue.

thanks!

should we update the echo statements in Makefile?

I wonder if we need to do this given that the lisp files have been moved to a directory in the root level. I'm syncing some of my branches with your master branch, so maybe I will figure this out in a few moments.

Integration with imenu

A simplistic imenu-create-index-function can look like this:

(defun tree-sitter-rust-imenu-index-function ()
  (thread-last (tree-sitter-query [(function_item (identifier) @function)])
    (seq-map (lambda (capture)
               (pcase-let ((`(_ . ,node) capture))
                 (cons (ts-node-text node)
                       (ts-node-start-position node)))))))

A more substantial implementation can use ts-query-matches to capture nested structures.

Support minor modes that rely on font-lock-mode

Currently tree-sitter-hl-mode disables all of font-lock-mode's highlighting.

It should instead disable only patterns set by the major mode, i.e. font-lock-defaults, and respect patterns added by minor modes and end users (through font-lock-add-keywords).

Note: This is a bit tricky, as the way font-lock-mode supports custom patterns is messy. See e.g. font-lock-refresh-defaults.

Set up CI to build a "universal" package

The universal package will contain the Lisp code, and 3 compiled dynamic libraries, one for each platform: macOS, Windows, Linux. This will allow it to be used almost anywhere, without requiring users to have compiler toolchains themselves.

If this is done well, it can be generalized into a pattern/tool for all future dynamic modules.

Note: Azure Pipelines support all 3 platforms.

Add equality check function

Equality check is important for syntax nodes.

Emacs's equal is broken for user-ptr. Two user pointer objects wrapping the same pointer and finalizer are not considered equal.

Integration with expand-region

The basic idea is simple: walking up the syntax tree from node at point.

Below is a simple implementation that expands region to the next bigger node:

(defun tree-sitter-mark-bigger-node ()
  (interactive)
  (let* ((p (point))
         (m (or (mark) p))
         (beg (min p m))
         (end (max p m))
         (root (ts-root-node tree-sitter-tree))
         (node (ts-get-descendant-for-position-range root beg end))
         (node-beg (ts-node-start-position node))
         (node-end (ts-node-end-position node)))
    ;; Node fits the region exactly. Try its parent node instead.
    (when (and (= beg node-beg) (= end node-end))
      (when-let ((node (ts-get-parent node)))
        (setq node-beg (ts-node-start-position node)
              node-end (ts-node-end-position node))))
    (set-mark node-end)
    (goto-char node-beg)))

(setq er/try-expand-list (append er/try-expand-list
                                 '(tree-sitter-mark-bigger-node)))

Conflict with evil-multiedit

Hi, I am using your amazing package from Doom Emacs and I found a conflict between tree-sitter-hl-mode and evil-multiedit.

To reproduce:

Select a few occurrences of a symbol with evil-multiedit-match-symbol-and-next, one occurrence in a string and one occurrence in a variable name
Try to edit them at once (for instance append some characters at the end) and it will not work.

I work with python and both packages are essential to my workflow! Please let me know if I can do anything to help you debug!

Cycling among different highlight files?

TLDR:

I've been working on highlights.scm files for tree-sitter-clojure and have begun to think that being able to cycle through some different highlighting schemes might be quite useful (please see below for background of the idea).

I scanned the current emacs lisp files a bit and the impression I got was that the idea is to have basically have one type of highlighting (possibly with customizations?). Does that seem correct?

If the idea of cycling through some different highlighting schemes sounds palatable, any suggestions / pointers on implementing such functionality?

Thanks for the consideration.

Background:

I watched a talk (https://youtu.be/l1b7Da2DnPo?t=831) by @tonsky recently where he discusses his alabaster theme: https://github.com/tonsky/sublime-scheme-alabaster

I implemented an approximation to it, gave it a try and found that it might be helpful for me sometimes (e.g. primarily when first examining unfamiliar code), but I doubt I would find it useful all of the time.

To give a few examples, it makes comments stand out, which I find helpful sometimes but not usually. It also does not highlight "built-in" functions / macros / special forms. When I am not yet familiar with a particular language, I would prefer to have such things highlighted and if possible be able to tell built-ins apart from "user-defined". Even if I am familiar, I thought I might be helped to be able to temporarily turn on such highlighting.

FWIW, the aforementioned repository has some details on the rationale behind some of his decisions and I found this to be interesting reading. I don't happen to agree with multiple issues, but I would guess that whatever I were to express as a list of important points, someone else would likely have their own differing opinion.

These experiences suggest to me that perhaps it would be useful to have a few different highlighting schemes one can switch among for different purposes. Perhaps one can think of them as different "glasses" for different occasions.

The query syntax seems to be evolving to make more things possible (e.g. possibly some form of not quantifier may show up at some point: tree-sitter/tree-sitter#705) and my experience with it so far suggests it may be easier to put together (as well as modify) different schemes than some previous valiant efforts (https://github.com/clojure-emacs/clojure-mode/blob/master/clojure-mode.el#L750-L926).

Mention how to install tree-sitter-cli in README

When I run ./bin/ensure-lang python, it returns an error

Running 'tree-sitter test' for python
./bin/ensure-lang: line 35: tree-sitter: command not found

And I went out to search how to install tree-sitter's cli.

It would be nice if you mention that tree-sitter-cli is required and how to install it.

Define what external dependencies are required

After installing Rust (with Rustup) and running './bin/build` I get this:

❯ ./bin/build
   Compiling emacs_module v0.10.0
error: failed to run custom build command for `emacs_module v0.10.0`

Caused by:
  process didn't exit successfully: `/home/jorge/code/emacs/emacs-tree-sitter/target/debug/build/emacs_module-a6842dec63007bb2/build-script-build` (exit code: 101)
--- stderr
./include/emacs-module.h:24:10: fatal error: 'stddef.h' file not found
./include/emacs-module.h:24:10: fatal error: 'stddef.h' file not found, err: true
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ()', src/libcore/result.rs:1084:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

It seems I don't have all dependencies on my Linux Mint 19.x (work laptop; I use Arch Linux at home and I'm more familiar with it), knowing what extra packages to install would be of use. I got Emacs from Snap

I have build-essential installed.

[question] future plans

After implementing highlights what are your plans for the emacs-tree-sitter?

I am particularly interested in implementing operations over the AST in the spirit of smartparens/paredit.

Have you thought about doing a presentation in emacsconf2020?

It'd be super cool to see this project shown there!
https://emacsconf.org/2020/cfp/

Cheers!

emacs-tree-sitter has issues with auto-yasnippet: wrong-type-argument char-or-string-p nil

After creating a snippet, when I expand the snippet in a Python buffer, I got this error:

Debugger entered--Lisp error: (wrong-type-argument char-or-string-p nil)
  tree-sitter--after-change(2045 2151 106)
  run-hook-with-args(tree-sitter--after-change 2045 2151 106)
  #f(compiled-function () #<bytecode 0x446903a9>)()
  funcall(#f(compiled-function () #<bytecode 0x446903a9>))
  (let nil (funcall (quote #f(compiled-function () #<bytecode 0x446903a9>))))
  eval((let nil (funcall (quote #f(compiled-function () #<bytecode 0x446903a9>)))))
  yas--snippet-create("if data.$1:\n            feed = team_pb2.UpdateTeamInput.$1.DESCRIPTOR.name\n            mask.paths.append(feed)" nil 2045 2045)
  yas-expand-snippet("if data.$1:\n            feed = team_pb2.UpdateTeamInput.$1.DESCRIPTOR.name\n            mask.paths.append(feed)")
  aya-expand()
  funcall-interactively(aya-expand)
  call-interactively(aya-expand nil nil)
  command-execute(aya-expand)

the result is that the string is expanded, but those parts with $1 are empty.

My Emacs version: GNU Emacs 26.3 (build 1, x86_64-apple-darwin18.2.0, NS appkit-1671.20 Version 10.14.3 (Build 18D109)) of 2019-09-02

Wrong highlighting of field declarations in c/c++

Hi!
In field declarations within structs/classes only the first field in comma separated lists of fields having common type gets assigned correct face (tree-sitter-hl-face:property.definition). The rest fields are assigned only (tree-sitter-hl-face:property). Also see screenshot below. I use f5065c8 with Emacs 27.1.

(require 'init-tree-sitter) error

when eval (require 'init-tree-sitter), it throw error below:

Debugger entered--Lisp error: (args-out-of-range #<buffer  *tar-data  *temp**> 513 16463596559)
  write-region(#<marker at 513 in  *tar-data  *temp**> 16463596559 "\037\213\010")
  tar-untar-buffer()
  (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer))
  (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  (catch (quote --cl-block-nil--) (if (string= version current-version) (if skip-if-installed (progn (message "tree-sitter-langs: Grammar bundle v%s was already installed; skipped" version) (throw (quote --cl-block-nil--) nil)) (message "tree-sitter-langs: Grammar bundle v%s was already installed; reinstalling" version)) (message "tree-sitter-langs: Installing grammar bundle v%s (was v%s)" version current-version)) (url-copy-file (tree-sitter-langs--bundle-url version os) bundle-file (quote ok-if-already-exists)) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))) (let ((temp-file tree-sitter-langs--bundle-version-file) (temp-buffer (get-buffer-create (generate-new-buffer-name " *temp file*")))) (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (let ((coding-system-for-write ...)) (insert version))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))) (if keep-bundle nil (delete-file bundle-file (quote trash))) (if (and (called-interactively-p (quote any)) (y-or-n-p (format "Show installed grammars in %s? " tree-sitter-langs--bin-dir))) (progn (save-current-buffer (set-buffer (find-file tree-sitter-langs--bin-dir)) (if (and (boundp (quote dired-omit-mode)) dired-omit-mode) (progn (dired-omit-mode -1)))))))
  (let* ((version (or version tree-sitter-langs--bundle-version)) (default-directory tree-sitter-langs--bin-dir) (bundle-file (tree-sitter-langs--bundle-file ".gz" version os)) (current-version (if (file-exists-p tree-sitter-langs--bundle-version-file) (progn (let ((temp-buffer ...)) (save-current-buffer (set-buffer temp-buffer) (unwind-protect ... ...))))))) (catch (quote --cl-block-nil--) (if (string= version current-version) (if skip-if-installed (progn (message "tree-sitter-langs: Grammar bundle v%s was already installed; skipped" version) (throw (quote --cl-block-nil--) nil)) (message "tree-sitter-langs: Grammar bundle v%s was already installed; reinstalling" version)) (message "tree-sitter-langs: Installing grammar bundle v%s (was v%s)" version current-version)) (url-copy-file (tree-sitter-langs--bundle-url version os) bundle-file (quote ok-if-already-exists)) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))) (let ((temp-file tree-sitter-langs--bundle-version-file) (temp-buffer (get-buffer-create (generate-new-buffer-name " *temp file*")))) (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (let (...) (insert version))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))) (if keep-bundle nil (delete-file bundle-file (quote trash))) (if (and (called-interactively-p (quote any)) (y-or-n-p (format "Show installed grammars in %s? " tree-sitter-langs--bin-dir))) (progn (save-current-buffer (set-buffer (find-file tree-sitter-langs--bin-dir)) (if (and (boundp ...) dired-omit-mode) (progn (dired-omit-mode -1))))))))
  tree-sitter-langs-install-grammars(:skip-if-installed)
  (if (bound-and-true-p tree-sitter-langs--testing) nil (tree-sitter-langs-install-grammars :skip-if-installed))
  (unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed))
  (progn (unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed)))
  eval((progn (unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed))) t)
  #f(compiled-function (&rest body) "Like `progn', but evaluates the body at compile time if you're compiling.\nThus, the result of the body appears to the compiler as a quoted\nconstant.  In interpreted code, this is entirely equivalent to\n`progn', except that the value of the expression may be (but is\nnot necessarily) computed at load time if eager macro expansion\nis enabled." #<bytecode 0x1a73d7>)((unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed)))
  (eval-when-compile (unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed)))
  eval-buffer(#<buffer  *load*-564722> nil "/home/kai/.emacs.d/lib/tree-sitter/langs/tree-sitter-langs.el" nil t)  ; Reading at buffer position 1671
  load-with-code-conversion("/home/kai/.emacs.d/lib/tree-sitter/langs/tree-sitter-langs.el" "/home/kai/.emacs.d/lib/tree-sitter/langs/tree-sitter-langs.el" nil t)
  #<subr require>(tree-sitter-langs)
  apply(#<subr require> tree-sitter-langs)
  require(tree-sitter-langs)

tree-sitter-hl-mode slows python-mode a little

Hi, thanks for this great package first of all.

In python-mode, when there are lots of thing to highlight (and maybe with long files like above 200), emacs is slower with tree-sitter-hl-mode. If I disable it, it speeds up. I can notice it when I hold C-n or C-p. Is it normal? I think tree-sitter supposed to be faster than font-lock.

Need help with C++ function indexing query

Experimenting with a drop in replacement for imenu indexing, per the other issue mentioning it. I'm not having much success getting a query capture back from the tools, and was hoping for some help.

Here's the query and command:

(defvar joe:func-query
  (ts-make-query (tree-sitter-require 'cpp) [(function_definition (identifier) @function)]))

(defun joe:make-index (capture)
  (pcase-let ((`(_ . ,node) capture))
    (cons (ts-node-text node)
      (ts-node-start-position node))))

(defun tree-sitter-c++-imenu-index-function ()
  (thread-last
    (ts-root-node tree-sitter-tree)
    (ts-query-captures joe:func-query)
    (seq-map #'joe:make-index)))

Running tree-sitter-debug-enable in the C++ file generates a full top-down tree, with plenty of function_definition nodes.

If I run a call to eval expression with the contents
(ts-query-captures joe:func-query (ts-root-node tree-sitter-tree))
it comes back with an empty vector, "[]"

Suggestions for debugging?

straight.el stuck at building tsc

with recent commits, emacs stuck at building tree-sitter -> building tsc. Old version works fine(without tsc)

emacs version: 28 native-comp
tree-sitter version: 2fa5045
system: x86_64 linux

my config is same as doc:

(straight-register-package
 '(tsc :host github
       :repo "ubolonton/emacs-tree-sitter"
       :files ("core/*.el")))
;; Base framework, syntax highlighting.
(straight-use-package
 '(tree-sitter :host github
               :repo "ubolonton/emacs-tree-sitter"
               :files ("lisp/*.el")))
;; Language bundle.
(straight-use-package
 '(tree-sitter-langs :host github
                     :repo "ubolonton/emacs-tree-sitter"
                     :files ("langs/*.el" "langs/queries")))

Replace grammar bundle with language-specific minor modes

Currently language grammars are bundled together, to be downloaded by tree-sitter-langs-install. However, each language has its own nuances that require additional customization. For each language, there should be a minor mode. Examples of things to be included in the minor mode:

Compiled grammar.
Syntax highlighting queries and additional faces.
imenu-create-index-function.
Custom code folding.
Custom evil text objects.

Implement syntax highlighting

References (using CSS selectors to assign syntax classes to nodes):

The implementation would likely be a replacement of font-lock-mode (and jit-lock), instead of being based on it.

Update 2019-10-05: Tree Query is tree-sitter's new pattern matching support, which enables a new approach that should supplant the above.

Find definitions/references

It would be nice if emacs-tree-sitter supported finding definitions and references of the symbol at point. It won't be as accurate as what language servers provide, but it should be faster. Finding references quickly would also enable highlighting of all references to the symbol at point.

I know that nvim-treesitter has a module that does this, but I'm hesitant to copy them since that would likely preclude inclusion of this package in GNU ELPA.

I could try implementing this myself if someone could direct me to the appropriate tree-sitter APIs. I'm guessing there's a better solution than naively traversing through the tree and checking identifiers. Upstream, HighlightConfiguration seems to have knowledge of local references, but if I'm not mistaken, I don't see any Emacs Lisp or Rust using it here in this package.

Crash when commenting code

I'm trying this out with tree-sitter-bash.

init.el:

(push "~/emacs-tree-sitter" load-path)
(require 'tree-sitter)
(add-hook 'sh-mode-hook (lambda ()
  (setq tree-sitter-language (ts-load-language "bash"))))

In this file:

ls

M-x tree-sitter-mode
Highlight the line
M-x comment-or-uncomment-region

Debugger entered--Lisp error: (args-out-of-range #<buffer a.sh> 1 6)
  ts-parse(#<user-ptr ptr=0x600000009be0 finalizer=0x10dd17c70> ts-buffer-input #<user-ptr ptr=0x60000000a3c0 finalizer=0x10dd17d30>)
  (setq tree-sitter-tree (ts-parse tree-sitter-parser (function ts-buffer-input) tree-sitter-tree))
  (let ((old-tree tree-sitter-tree)) (setq tree-sitter-tree (ts-parse tree-sitter-parser (function ts-buffer-input) tree-sitter-tree)) (run-hook-with-args (quote tree-sitter-after-change-functions) old-tree))
  tree-sitter--do-parse()
  (progn (ts-edit-tree tree-sitter-tree tree-sitter--start-byte tree-sitter--old-end-byte tree-sitter--new-end-byte tree-sitter--start-point tree-sitter--old-end-point tree-sitter--new-end-point) (tree-sitter--do-parse))
  (if tree-sitter-tree (progn (ts-edit-tree tree-sitter-tree tree-sitter--start-byte tree-sitter--old-end-byte tree-sitter--new-end-byte tree-sitter--start-point tree-sitter--old-end-point tree-sitter--new-end-point) (tree-sitter--do-parse)))
  tree-sitter--after-change(1 3 0)
  comment-region-internal(1 3 "# " nil nil nil nil nil t)
  comment-region-default(1 3 nil)
  comment-region(1 3 nil)
  comment-or-uncomment-region(1 3 nil)
  funcall-interactively(comment-or-uncomment-region 1 3 nil)
  call-interactively(comment-or-uncomment-region record nil)
  command-execute(comment-or-uncomment-region record)
  execute-extended-command(nil "comment-or-uncomment-region" "comment-o")
  funcall-interactively(execute-extended-command nil "comment-or-uncomment-region" "comment-o")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

installation guide for package management other than package.el

First of all, thanks for the great work!

I'm using straight.el as alternative to package.el to manage my package. I had a hard time compile and install emacs-tree-sitter. I was wondering if you can provide a manual install guide.

Add support for testing highlighting using Sublime's test format

Tree-sitter itself supports this tree-sitter/tree-sitter#499.

Intelligent cursor restoration after auto-formatting

I've never used tree-sitter so let me know if this is not possible or doesn't make much sense.

Restoring buffer position after applying styling fixes can be difficult. My frequent use case is with the black formatter using https://github.com/lassik/emacs-format-all-the-code on save.

Black makes a guarantee that the AST is unmodified after formatting, so it seems to me that combining tree-sitter with this the user could be placed on the same tree node as they were before the format. This would make it so that autoformatting would leave your cursor in the same logical place as it was before, even if the number of lines of code was radically changed.

Consider converting the point in the library

ATM the offset has to be 1+ or 1- because tree sitter is zero based while emacs 1 based. IMO it will be better to fix that now in the tree-sitter library until there aren't a lot of clients of emacs tree-sitter bindings.

Doesn't compile on Rust 1.47

On Rust 1.47, compilation fails with a lot of errors similar to this:

error[E0423]: expected value, found macro `env`
   --> core/src/lang.rs:182:52
    |
182 |     "lang-node-type-named-p" fn node_kind_is_named(type_id: u16) -> bool
    |                                                    ^^^^^^^ not a value
    |
help: use `!` to invoke the macro
    |
182 |     "lang-node-type-named-p" fn node_kind_is_named(type_id!: u16) -> bool
    |                                                           ^
help: consider importing this function instead
    |
1   | use crate::lang::os::unix::process::sys::os::env;
    |

How to make spell-fu work with tree-sitter-hl?

If I enable tree-sitter-hl-mode in a buffer, it gets highlighted correctly and my TODO and NOTE items still get highlighted (by the hl-todo minor mode) but the misspelled words in the buffer lose their red underline from spell-fu-mode. Related to #45?

solution for multi-language files

@yyoncho noted that in Emacs sometimes a mapping of major mode and language is not as straightforward as N -> 1, but rather N -> N, that is to say that some major modes have to handle different languages at once, like web-mode which handles HTML, CSS, TypeScript, JavaScript and so on.

Thus, is required that emacs-tree-sitter handles this kind of edge cases gracefully.

Write a guide doc

Tree-sitter itself has good documentation, but its target audience is different. The guide doc should:

Leave out details irrelevant to writing Emacs packages.
Explain Emacs-specific nuances.
Provide examples in Lisp.

Install grammar binaries relative to user-emacs-directory, instead of using locate-library

Many emacs packages, e.g. lsp-mode, install binaries relative to ~/.emacs.d and allow the binary location to be determined through a variable.

In contrast, this package installs its non-elisp data relative to the compiled library (via load-library). This breaks when using gccemacs, because gccemacs compiles files into their own directory.

If you used the convention and installed into ~/.emacs.d/ , or at least allowed the user to control where the data was installed, it would fix this problem.

Thank you!

Two tree-sitter.el implementations

@ubolonton and @karlotness are you aware of each other's tree-sitter.el implementations?

This is going to be a problem once you both want to get your implementations added to GNU Elpa or Melpa.

I haven't a concrete proposal, but suggest that the two of you talk about this. Oh, and you might want to mention the other implementation in your README's and if you already had such a talk, then also link to that.

file-missing "Cannot open load file" "file or directory not found" "tree-sitter-dyn"

after successfully building the bindings and adding the repository to my load-path and evaluating (require 'tree-sitter) I get the following error:

Debugger entered--Lisp error: (file-missing "Cannot open load file" "No existe el archivo o el directorio" "tree-sitter-dyn")
  require(tree-sitter-dyn)
  eval-buffer(#<buffer  *load*-454245> nil "/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter-core.el" nil t)  ; Reading at buffer position 306
  load-with-code-conversion("/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter-core.el" "/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter-core.el" nil t)
  require(tree-sitter-core)
  eval-buffer(#<buffer  *load*> nil "/home/jorge/code/emacs/emacs-tree-sitter/tree-sitt..." nil t)  ; Reading at buffer position 778
  load-with-code-conversion("/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter.el" "/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter.el" nil t)
  require(tree-sitter)
  (progn (require 'tree-sitter))
  eval((progn (require 'tree-sitter)) t)
  elisp--eval-last-sexp(nil)
  eval-last-sexp(nil)
  funcall-interactively(eval-last-sexp nil)
  call-interactively(eval-last-sexp nil nil)
  command-execute(eval-last-sexp)

This is the content of the root of the project, locally:

❯ tree -a -L 1
.
├── .azure-pipelines
├── bin
├── Cargo.lock
├── Cargo.toml
├── Cask
├── .git
├── .gitignore
├── grammars
├── README.md
├── src
├── target
├── .travis.yml
├── tree-sitter-core.el
├── tree-sitter-debug.el
├── tree-sitter-dyn.so
├── tree-sitter.el
└── tree-sitter-tests.el

6 directories, 11 files

No support for rustic-mode

I use rustic-mode instead of rust-mode as the major mode for Rust files in emacs and emacs-tree-sitter has no support for it :(

Can you please add support for rustic-mode

syntax highlighting for c++ removes font-lock-variable-name-face

currently syntax highlighting in c++ is too much. When everything is highlighted on the page nothing becomes discernible. Currently I need to set tree-sitter-hl-face:variable to inherit from default just to regain some sanity. But then I lose variable name highlighting (during a variables declaration that is). How can I retain some features from the default font lock (such as font-lock-variable-name-face) without turning off tree-sitter?

describe further goals in issues or milestones

I['m entirely out of my depth here, as I have only the most minimal understanding of tree-sitter and the approach you're taking here. But I wonder if you could add some issues to track (even aspirational) goals for hte project beyond syntax highlighting. For instance, in the readme you talk about these features:

More flexible code folding.
Structural editing (like Paredit, or even better) for non-Lisp code.
More informative indexing for imenu.

Maybe they don't actually belong in this package, but in some extensions (esp since people have their own idiosyncratic preferences for things like lispy/paredit and code folding); but it would be pretty nice to see a bit more fleshed out ideas of how these features would work, and what's in principle possible now, etc etc.

Really excited to see this project -- fast and perfect syntax highlighting on its own will be a huge win, and if structural editing in all languages comes around too, it will be fantastic.

rainbow-delimiters stops working when tree-sitter-hl-mode is enabled

Steps to reproduce are quite simple:

open any file (whose major mode is supported for tree-sitter)
Enable tree-sitter-mode and tree-sitter-hl-mode
Run rainbow-delimiters-mode

You can see that none of the brackets will be highlighted.

Error reporting for ts-make-query is confusing

When a query is invalid, ts-make-query raises an error like this:

(rust-panic "called `Result::unwrap()` on an `Err` value: Syntax(1, \"[(call_expression function: (identifier) @function)]\\n^\")")

This is because _make_query uses unwrap.

It should instead define a new error class, e.g. ts-query-error, and raise errors with good messages.

Automatic f-strings

Let me know if this isn't an appropriate place to file an issue, but I'm trying to remember elisp and wanted to clone a new PyCharm feature:

Forgot to add ‘f’ to your f-string? PyCharm now auto-enables f-strings when the user adds curly braces within a string statement

So foo = '{' gets changed to foo = f'{'.

I came up with what is likely a very brittle solution and was wondering if you had any advice:

;; todo: run after character insert, maybe tree-sitter-after-change-functions?
(defun maybe-convert-string-to-f-string()
  (if (= ?{ (char-after))
      (make-node-f-string (tree-sitter-node-at-point))))

(defun make-node-f-string(node)
  ;; todo: should this query object be made outside of the function for performance?
  (let ((is-string-query (ts-make-query (tree-sitter-require 'python) "(string) @string")))
    ;; todo: text-function seems extraneous, is there a better way to just check if the node i'm
    ;; on is a string, returning t or nil?
    (when (ts-query-matches is-string-query node 'ts--buffer-substring-no-properties)
      (save-excursion
        ;; todo: is there a tree-sitter way of replacing a node?
        (goto-char (ts-node-start-position node))
        (insert "f")))))

Modifying tree-sitter-langs-repos doesn't download and build new additions

Perhaps I'm not groking the README right, but after modifying tree-sitter-langs-repos and running make ensure/rust, I just get the shared libraries that are located in the original bundle. Do I have to add this grammar to your bintray repo?

Steps to reproduce

Clone emacs-tree-sitter
Append (hit "bdeac01") to the end of tree-sitter-langs-repos
Run make ensure/rust.
Check ./langs/bin for hit.dylib

Submit to GNU ELPA

Hi. Thanks for this; it sounds neat.

It aims to be the foundation for a new breed of Emacs packages that understand code structurally.

TODOs
Submit to MELPA.

With the above in mind, please do consider putting this in GNU ELPA so that all package authors can use it? It would be a shame to limit its usefulness as a dependency to MELPA packages only.

Potential namespace conflict with ts.el

Hi @ubolonton,

Following the recent discussions on emacs-devel, I saw you mention a ts-parse function. I can't find that function's definition in the master branch, but I do see some calls to it. So I thought I should mention that there is a potential namespace conflict with my library https://github.com/alphapapa/ts.el, which implements a ts-parse function.

Thanks for your work on this package. I'm looking forward to the improvements it will bring to Emacs.

Export org as html file with syntax highlighting error via htmlize and emacs-tree-sitter

Reproduce the problem with the smallest example

There is the init.el file below.
And run emacs -Q -l init.el to set up the environment for reproducing the problem.

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))
(straight-use-package 'use-package)
(setq-default use-package-always-defer t)

(use-package htmlize
   :straight t)

(use-package tree-sitter
  :straight t
  :hook
  ((python-mode . tree-sitter-hl-mode)))

(use-package tree-sitter-langs
  :straight t
  :demand)

The test file test.org.

#+BEGIN_SRC python
  def hello(name):
      return f"hello {name}"

  hello("world")
#+END_SRC

Run emacs -Q -l init.el test.org and input <C-c C-e h h> to export org file as HTML file.
But it produces an error jit-lock-fontify-now: Wrong type argument: number-or-marker-p, nil

run with toggle-debug-on-error to get the backtrace of this error.

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  jit-lock--run-functions(1 60)
  jit-lock-fontify-now(1 60)
  org-font-lock-ensure()
  org-html-fontify-code("def hello(name):\n    return f\"hello {name}\"\n\nhello..." "python")
  org-html-do-format-code("def hello(name):\n    return f\"hello {name}\"\n\nhello..." "python" nil t nil nil)
  org-html-format-code((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    retur..." :post-blank 0 :post-affiliated 1 ...)) (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  org-html-src-block((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    retur..." :post-blank 0 :post-affiliated 1 ...)) nil (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  org-export-data((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    retur..." :post-blank 0 :post-affiliated 1 ...)) (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  #f(compiled-function (element) #<bytecode 0x1726fed925d5f7c7>)((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    return f\"hello {name}\"\n\nhello..." :post-blank 0 :post-affiliated 1 :parent (section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil #31)) #1))))
  mapconcat(#f(compiled-function (element) #<bytecode 0x1726fed925d5f7c7>) ((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    return f\"hello {name}\"\n\nhello..." :post-blank 0 :post-affiliated 1 :parent (section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil #33)) . #2)))) "")
  org-export-data((section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil #1)) (src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    retur..." :post-blank 0 :post-affiliated 1 ...))) (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  #f(compiled-function (element) #<bytecode 0x1726fed925d5f7c7>)((section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil #1)) (src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    return f\"hello {name}\"\n\nhello..." :post-blank 0 :post-affiliated 1 :parent #1))))
  mapconcat(#f(compiled-function (element) #<bytecode 0x1726fed925d5f7c7>) ((section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil . #2)) (src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    return f\"hello {name}\"\n\nhello..." :post-blank 0 :post-affiliated 1 :parent #3)))) "")
  org-export-data((org-data nil (section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent #1) (src-block ...))) (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  org-export-as(html nil nil nil (:output-file "test.html"))
  org-export-to-file(html "test.html" nil nil nil nil nil)
  org-html-export-to-html(nil nil nil nil)
  org-export-dispatch(nil)
  funcall-interactively(org-export-dispatch nil)
  call-interactively(org-export-dispatch nil nil)
  command-execute(org-export-dispatch)

It will fix this problem but will export an HTML file without the syntax highlighting if I disable the htmlize or emacs-tree-sitter.

Tree visualization using treemacs

This would be useful as an exploration tool, and a visual aid in building queries for other features.

See https://github.com/Alexander-Miller/treemacs/blob/master/Extensions.org.

Integrating tree-sitter-mode directly into a new major-mode

I'm writing a new major mode for WebAssembly text formats and would like to use the WebAssembly tree-sitter grammar (here) for highlighting.

Since this is a new mode being written from scratch, it sounds like it might be better to integrate support for the tree-sitter mode directly into this new major mode, rather than including the queries and other definitions in this repository.

However, I'm not quite sure what I need to do in order to build the tree-sitter functionality in this new mode. The documentation doesn't seem to have much to say about that.

Is it currently feasible to include support directly into a new mode? If so, can you give a description for how I need to set things up for that?

ruby: identifiers mis-highlighted as "function.method"

The below ruby code:

foo = 123

Gives foo a face of tree-sitter-hl-face:function.method which isn't correct (it should be variable or similar). I poked around a bit and my guess would be the ((identifier) @function.method (#is-not? local)) highlight rule is matching too much suggesting the locals stuff isn't working (?).

I'm guessing it is an issue here vs tree-sitter-ruby, but I'm really not sure.

emacs-tree-sitter / elisp-tree-sitter Goto Github PK

elisp-tree-sitter's Issues

Reproduce the problem with the smallest example

Recommend Projects

Recommend Topics

Recommend Org

Jobs