Github Help home page Github Help logo

elisp-tree-sitter's Introduction

ELisp Tree-sitter

Documentation Azure Pipelines GitHub Actions

This is an Emacs Lisp binding for tree-sitter, an incremental parsing library. It requires Emacs 25.1 or above, built with dynamic module support.

It aims to be the foundation for a new breed of Emacs packages that understand code structurally. For example:

  • Faster, fine-grained code highlighting.
  • More flexible code folding.
  • Structural editing (like Paredit, or even better) for non-Lisp code.
  • More informative indexing for imenu.

The author of tree-sitter articulated its merits a lot better in this Strange Loop talk.

Installation

See the installation section in the documentation.

If you want to hack on emacs-tree-sitter itself, see the next section instead.

Setup for Development

Note: On Windows, use Powershell instead of Bash or cmd.exe.

  • Clone this repo with the --recursive flag.
  • Add 3 of its directories to load-path: core/, lisp/ and langs/.
  • Install cask.
  • Run bin/setup.

If you want to hack on the high-level features (in Lisp) only:

  • Make changes to the .el files.
  • Add tests to tree-sitter-tests.el and run them with bin/test.

If you want to build additional (or all) grammars from source, or work on the core dynamic module, see the next 2 sections.

Building grammars from source

Note: If you are only interested in building the grammar binaries, not the dynamic module, check out tree-sitter-langs.

  • Install NodeJS. It is needed to generate the grammar code from the JavaScript DSL. The recommended tool to manage NodeJS is volta.
  • Install tree-sitter CLI tool: (Its binary can also be downloaded directly from GitHub.) Note: version 0.20+ cannot be used, as they introduced a breaking change in binary storage location.
    # For yarn user
    yarn global add tree-sitter-cli@0.19.3
    
    # For npm user
    npm install -g tree-sitter-cli@0.19.3
    
  • Run langs/script/compile. For example:
    langs/script/compile rust
    

Working on the dynamic module

  • Install the Rust toolchain.
  • Install clang, to generate the raw Rust binding for emacs-module.h. On Windows, set LIBCLANG_PATH as well. For example:
    $env:LIBCLANG_PATH = "$(scoop prefix llvm)\bin"
    
  • Build:
    bin/build
    
  • Test:
    bin/test
    
  • Continuously rebuild and test on change (requires cargo-watch):
    bin/test watch
    

To test against a different version of Emacs, set the environment variable EMACS (e.g. EMACS=/snap/bin/emacs make test).

Alternative

Binding through C instead of Rust: https://github.com/karlotness/tree-sitter.el

Contribution

Contributions are welcomed. Please take a look at the issue list for ideas, or create a new issue to describe any idea you have for improvement.

For language-specific issues/features, please check out tree-sitter-langs instead.

Show respect and empathy towards others. Both technical empathy and general empathy are highly valued.

Contributors

ubolonton avatar shackra avatar jcs090218 avatar tarsius avatar theothornhill avatar rolandwalker avatar kiennq avatar meain avatar davidpa9708 avatar fbergroth avatar iwahbe avatar wyuenho avatar mplamann avatar theHamsta avatar Vincent-P avatar sogaiu avatar

Stargazers

 avatar jdhao avatar driftcrow avatar AM avatar Jon Rostran avatar Samuel Matias avatar Owain West avatar  avatar  avatar Kon Tsaki avatar  avatar SadSock avatar Leon Ferreira Bellini avatar Jörn Gersdorf avatar Dmitry Kosenkov avatar Konstantin Kharlamov avatar 子龙山人 avatar Rice Wine avatar Thomas Wu avatar  avatar Novak Boškov avatar Valer Cara avatar  avatar Alberto Gireud avatar Rainer Poisel avatar  avatar Dixon Sean Low Yan Feng avatar Harry Fei avatar ZhangJun avatar  avatar Ozan Şener avatar  avatar Simon Pugnet avatar Vojtěch Balák avatar Michael Eliachevitch avatar Bram avatar Stanley Chan avatar Evan avatar Bastien Rivière avatar Yi Liu avatar  avatar Mikhail Glushenkov avatar Wei Zhang avatar Sidharth Arya avatar  avatar Ye Zhaoliang avatar Jumpei KAWAMI avatar t. m. k. avatar Zitian Wang avatar  avatar Thang Pham avatar  avatar Brendan avatar Arthur Colombini Gusmão avatar Aleksandar Dimitrov avatar Hector Orellana avatar Fausto Marques Pinheiro Junior avatar Vincent Laporte avatar Arne Bahlo avatar Max Tweddell avatar Jake Moss avatar  avatar Tim Schubert avatar Abin Simon avatar  avatar John Hamelink avatar Masanori Ogino avatar  avatar Lida avatar Andrew Hlynskyi avatar Yigit Emre Sahinoglu avatar Will Bush avatar Laz avatar Daanturo avatar Tim Loderhose avatar Jacksoncy avatar Sergey Petrov avatar Rik Chilvers avatar  avatar  avatar Eason Huang avatar Sasha Wasilewska avatar Szabó Krisztián avatar Dušan avatar Justin Garcia avatar Matthew Avant avatar Sadie L. Bartholomew avatar Alexey Shiklomanov avatar  avatar Pavel Atanasov avatar  avatar Bruno Coimbra avatar Liu Hui avatar Justin Heyes-Jones avatar Ben Lawson avatar Tristan de Cacqueray avatar  avatar Zen Monk Alain M. Lafon avatar Francesco Noacco avatar George Plymale II avatar

Watchers

Abhik Khanra avatar Tuấn-Anh Nguyễn avatar Thomas Hisch avatar  avatar  avatar Alan Zimmerman avatar Yang Chuang avatar  avatar Peter avatar Jorge Araya avatar Stanley Chan avatar Fredrik Bergroth avatar nth avatar M. Tong avatar Sourabh Cheedella avatar xd1le avatar  avatar Daniel Swanson avatar Roman Rudakov avatar Tiancheng-Luo avatar Stephan Seitz avatar Jen-Chieh Shen avatar  avatar Ivan Yonchovski avatar  avatar  avatar  avatar  avatar

elisp-tree-sitter's Issues

solution for multi-language files

@yyoncho noted that in Emacs sometimes a mapping of major mode and language is not as straightforward as N -> 1, but rather N -> N, that is to say that some major modes have to handle different languages at once, like web-mode which handles HTML, CSS, TypeScript, JavaScript and so on.

Thus, is required that emacs-tree-sitter handles this kind of edge cases gracefully.

Submit to GNU ELPA

Hi. Thanks for this; it sounds neat.

It aims to be the foundation for a new breed of Emacs packages that understand code structurally.

TODOs
Submit to MELPA.

With the above in mind, please do consider putting this in GNU ELPA so that all package authors can use it? It would be a shame to limit its usefulness as a dependency to MELPA packages only.

Upcoming changes to tree-sitter query syntax

Hey there 👋

I’m filing this issue to make y’all aware that there are some upcoming changes to tree-sitter’s query syntax, stemming from tree-sitter/tree-sitter#615. The implementation of these changes should retain backwards compatibility, but going forward we’d recommend using the new syntax, as it’s (in our opinion!) more readable and powerful. We’ll be updating the highlighting.scm queries in the various tree-sitter language packages, so if you depend directly on them via submodules or the like, everything should Just Work.

If this is going to cause any grief for this project, please reach out on the tree-sitter issues board and let us know, and we’ll do our best to make the transition easy for you.

Thank you very much for all your work on this project: as an Emacs devotee, I personally am hugely excited about the prospect of using tree-sitter to power my Emacs!

cc @maxbrunsfeld

Automatic f-strings

Let me know if this isn't an appropriate place to file an issue, but I'm trying to remember elisp and wanted to clone a new PyCharm feature:

Forgot to add ‘f’ to your f-string? PyCharm now auto-enables f-strings when the user adds curly braces within a string statement

So foo = '{' gets changed to foo = f'{'.

I came up with what is likely a very brittle solution and was wondering if you had any advice:

;; todo: run after character insert, maybe tree-sitter-after-change-functions?
(defun maybe-convert-string-to-f-string()
  (if (= ?{ (char-after))
      (make-node-f-string (tree-sitter-node-at-point))))

(defun make-node-f-string(node)
  ;; todo: should this query object be made outside of the function for performance?
  (let ((is-string-query (ts-make-query (tree-sitter-require 'python) "(string) @string")))
    ;; todo: text-function seems extraneous, is there a better way to just check if the node i'm
    ;; on is a string, returning t or nil?
    (when (ts-query-matches is-string-query node 'ts--buffer-substring-no-properties)
      (save-excursion
        ;; todo: is there a tree-sitter way of replacing a node?
        (goto-char (ts-node-start-position node))
        (insert "f")))))

Replace grammar bundle with language-specific minor modes

Currently language grammars are bundled together, to be downloaded by tree-sitter-langs-install. However, each language has its own nuances that require additional customization. For each language, there should be a minor mode. Examples of things to be included in the minor mode:

  • Compiled grammar.
  • Syntax highlighting queries and additional faces.
  • imenu-create-index-function.
  • Custom code folding.
  • Custom evil text objects.

Integrating tree-sitter-mode directly into a new major-mode

I'm writing a new major mode for WebAssembly text formats and would like to use the WebAssembly tree-sitter grammar (here) for highlighting.

Since this is a new mode being written from scratch, it sounds like it might be better to integrate support for the tree-sitter mode directly into this new major mode, rather than including the queries and other definitions in this repository.

However, I'm not quite sure what I need to do in order to build the tree-sitter functionality in this new mode. The documentation doesn't seem to have much to say about that.

Is it currently feasible to include support directly into a new mode? If so, can you give a description for how I need to set things up for that?

Set up CI to build a "universal" package

The universal package will contain the Lisp code, and 3 compiled dynamic libraries, one for each platform: macOS, Windows, Linux. This will allow it to be used almost anywhere, without requiring users to have compiler toolchains themselves.

If this is done well, it can be generalized into a pattern/tool for all future dynamic modules.

Note: Azure Pipelines support all 3 platforms.

How to make spell-fu work with tree-sitter-hl?

If I enable tree-sitter-hl-mode in a buffer, it gets highlighted correctly and my TODO and NOTE items still get highlighted (by the hl-todo minor mode) but the misspelled words in the buffer lose their red underline from spell-fu-mode. Related to #45?

Export org as html file with syntax highlighting error via htmlize and emacs-tree-sitter

Reproduce the problem with the smallest example

There is the init.el file below.
And run emacs -Q -l init.el to set up the environment for reproducing the problem.

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))
(straight-use-package 'use-package)
(setq-default use-package-always-defer t)

(use-package htmlize
   :straight t)

(use-package tree-sitter
  :straight t
  :hook
  ((python-mode . tree-sitter-hl-mode)))

(use-package tree-sitter-langs
  :straight t
  :demand)

The test file test.org.

#+BEGIN_SRC python
  def hello(name):
      return f"hello {name}"

  hello("world")
#+END_SRC

Run emacs -Q -l init.el test.org and input <C-c C-e h h> to export org file as HTML file.
But it produces an error jit-lock-fontify-now: Wrong type argument: number-or-marker-p, nil

run with toggle-debug-on-error to get the backtrace of this error.

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  jit-lock--run-functions(1 60)
  jit-lock-fontify-now(1 60)
  org-font-lock-ensure()
  org-html-fontify-code("def hello(name):\n    return f\"hello {name}\"\n\nhello..." "python")
  org-html-do-format-code("def hello(name):\n    return f\"hello {name}\"\n\nhello..." "python" nil t nil nil)
  org-html-format-code((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    retur..." :post-blank 0 :post-affiliated 1 ...)) (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  org-html-src-block((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    retur..." :post-blank 0 :post-affiliated 1 ...)) nil (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  org-export-data((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    retur..." :post-blank 0 :post-affiliated 1 ...)) (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  #f(compiled-function (element) #<bytecode 0x1726fed925d5f7c7>)((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    return f\"hello {name}\"\n\nhello..." :post-blank 0 :post-affiliated 1 :parent (section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil #31)) #1))))
  mapconcat(#f(compiled-function (element) #<bytecode 0x1726fed925d5f7c7>) ((src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    return f\"hello {name}\"\n\nhello..." :post-blank 0 :post-affiliated 1 :parent (section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil #33)) . #2)))) "")
  org-export-data((section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil #1)) (src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    retur..." :post-blank 0 :post-affiliated 1 ...))) (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  #f(compiled-function (element) #<bytecode 0x1726fed925d5f7c7>)((section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil #1)) (src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    return f\"hello {name}\"\n\nhello..." :post-blank 0 :post-affiliated 1 :parent #1))))
  mapconcat(#f(compiled-function (element) #<bytecode 0x1726fed925d5f7c7>) ((section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent (org-data nil . #2)) (src-block (:language "python" :switches nil :parameters nil :begin 1 :end 90 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "def hello(name):\n    return f\"hello {name}\"\n\nhello..." :post-blank 0 :post-affiliated 1 :parent #3)))) "")
  org-export-data((org-data nil (section (:begin 1 :end 90 :contents-begin 1 :contents-end 90 :post-blank 0 :post-affiliated 1 :parent #1) (src-block ...))) (:export-options nil :back-end #s(org-export-backend :name html :parent nil :transcoders (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :options (... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) :filters (... ... ...) :blocks nil :menu (104 "Export to HTML" ...)) :translate-alist ((bold . org-html-bold) (center-block . org-html-center-block) (clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) (dynamic-block . org-html-dynamic-block) (entity . org-html-entity) (example-block . org-html-example-block) (export-block . org-html-export-block) (export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) (footnote-definition . org-html-footnote-definition) (footnote-reference . org-html-footnote-reference) (headline . org-html-headline) (horizontal-rule . org-html-horizontal-rule) (inline-src-block . org-html-inline-src-block) (inlinetask . org-html-inlinetask) (inner-template . org-html-inner-template) (italic . org-html-italic) (item . org-html-item) (keyword . org-html-keyword) (latex-environment . org-html-latex-environment) (latex-fragment . org-html-latex-fragment) (line-break . org-html-line-break) (link . org-html-link) (node-property . org-html-node-property) ...) :exported-data #<hash-table eq 0/4001 0x1ff133784a4f> :input-buffer "test.org" :input-file "/Users/linw1995/Documents/..." :html-doctype "xhtml-strict" :html-container "div" :description nil :keywords nil :html-html5-fancy nil :html-link-use-abs-url nil :html-link-home "" ...))
  org-export-as(html nil nil nil (:output-file "test.html"))
  org-export-to-file(html "test.html" nil nil nil nil nil)
  org-html-export-to-html(nil nil nil nil)
  org-export-dispatch(nil)
  funcall-interactively(org-export-dispatch nil)
  call-interactively(org-export-dispatch nil nil)
  command-execute(org-export-dispatch)

It will fix this problem but will export an HTML file without the syntax highlighting if I disable the htmlize or emacs-tree-sitter.

file-missing "Cannot open load file" "file or directory not found" "tree-sitter-dyn"

after successfully building the bindings and adding the repository to my load-path and evaluating (require 'tree-sitter) I get the following error:

Debugger entered--Lisp error: (file-missing "Cannot open load file" "No existe el archivo o el directorio" "tree-sitter-dyn")
  require(tree-sitter-dyn)
  eval-buffer(#<buffer  *load*-454245> nil "/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter-core.el" nil t)  ; Reading at buffer position 306
  load-with-code-conversion("/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter-core.el" "/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter-core.el" nil t)
  require(tree-sitter-core)
  eval-buffer(#<buffer  *load*> nil "/home/jorge/code/emacs/emacs-tree-sitter/tree-sitt..." nil t)  ; Reading at buffer position 778
  load-with-code-conversion("/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter.el" "/home/jorge/code/emacs/emacs-tree-sitter/tree-sitter.el" nil t)
  require(tree-sitter)
  (progn (require 'tree-sitter))
  eval((progn (require 'tree-sitter)) t)
  elisp--eval-last-sexp(nil)
  eval-last-sexp(nil)
  funcall-interactively(eval-last-sexp nil)
  call-interactively(eval-last-sexp nil nil)
  command-execute(eval-last-sexp)

This is the content of the root of the project, locally:

❯ tree -a -L 1
.
├── .azure-pipelines
├── bin
├── Cargo.lock
├── Cargo.toml
├── Cask
├── .git
├── .gitignore
├── grammars
├── README.md
├── src
├── target
├── .travis.yml
├── tree-sitter-core.el
├── tree-sitter-debug.el
├── tree-sitter-dyn.so
├── tree-sitter.el
└── tree-sitter-tests.el

6 directories, 11 files

Syntax highlighting doesn't work for java annotations with scoped_identifier

To reproduce:

  1. Create java class:
@Config.Source("any value")
public class Test {
  
}
  1. Add the following to init.el:
(tree-sitter-hl-add-patterns 'java
    [
     ;; Scoped annotations
     (annotation
      (scoped_identifier) @parameter)

     (marker_annotation
      (scoped_identifier) @parameter)
     ])

Expected result:

  • Annotation highlighted using tree-sitter-hl-face:parameter

Actual result:

  • Config part highlighted using tree-sitter-hl-face:type face and .Source part highlighted using default face.

Emacs version:
GNU Emacs 27.1 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw3d scroll bars) of 2020-08-11
tree-sitter-bug

Integration with expand-region

The basic idea is simple: walking up the syntax tree from node at point.

Below is a simple implementation that expands region to the next bigger node:

(defun tree-sitter-mark-bigger-node ()
  (interactive)
  (let* ((p (point))
         (m (or (mark) p))
         (beg (min p m))
         (end (max p m))
         (root (ts-root-node tree-sitter-tree))
         (node (ts-get-descendant-for-position-range root beg end))
         (node-beg (ts-node-start-position node))
         (node-end (ts-node-end-position node)))
    ;; Node fits the region exactly. Try its parent node instead.
    (when (and (= beg node-beg) (= end node-end))
      (when-let ((node (ts-get-parent node)))
        (setq node-beg (ts-node-start-position node)
              node-end (ts-node-end-position node))))
    (set-mark node-end)
    (goto-char node-beg)))

(setq er/try-expand-list (append er/try-expand-list
                                 '(tree-sitter-mark-bigger-node)))

Add equality check function

Equality check is important for syntax nodes.

Emacs's equal is broken for user-ptr. Two user pointer objects wrapping the same pointer and finalizer are not considered equal.

Modifying tree-sitter-langs-repos doesn't download and build new additions

Perhaps I'm not groking the README right, but after modifying tree-sitter-langs-repos and running make ensure/rust, I just get the shared libraries that are located in the original bundle. Do I have to add this grammar to your bintray repo?

Steps to reproduce

  1. Clone emacs-tree-sitter
  2. Append (hit "bdeac01") to the end of tree-sitter-langs-repos
  3. Run make ensure/rust.
  4. Check ./langs/bin for hit.dylib

ruby: identifiers mis-highlighted as "function.method"

The below ruby code:

foo = 123

Gives foo a face of tree-sitter-hl-face:function.method which isn't correct (it should be variable or similar). I poked around a bit and my guess would be the ((identifier) @function.method (#is-not? local)) highlight rule is matching too much suggesting the locals stuff isn't working (?).

I'm guessing it is an issue here vs tree-sitter-ruby, but I'm really not sure.

describe further goals in issues or milestones

I['m entirely out of my depth here, as I have only the most minimal understanding of tree-sitter and the approach you're taking here. But I wonder if you could add some issues to track (even aspirational) goals for hte project beyond syntax highlighting. For instance, in the readme you talk about these features:

  • More flexible code folding.
  • Structural editing (like Paredit, or even better) for non-Lisp code.
  • More informative indexing for imenu.

Maybe they don't actually belong in this package, but in some extensions (esp since people have their own idiosyncratic preferences for things like lispy/paredit and code folding); but it would be pretty nice to see a bit more fleshed out ideas of how these features would work, and what's in principle possible now, etc etc.

Really excited to see this project -- fast and perfect syntax highlighting on its own will be a huge win, and if structural editing in all languages comes around too, it will be fantastic.

Find definitions/references

It would be nice if emacs-tree-sitter supported finding definitions and references of the symbol at point. It won't be as accurate as what language servers provide, but it should be faster. Finding references quickly would also enable highlighting of all references to the symbol at point.

I know that nvim-treesitter has a module that does this, but I'm hesitant to copy them since that would likely preclude inclusion of this package in GNU ELPA.

I could try implementing this myself if someone could direct me to the appropriate tree-sitter APIs. I'm guessing there's a better solution than naively traversing through the tree and checking identifiers. Upstream, HighlightConfiguration seems to have knowledge of local references, but if I'm not mistaken, I don't see any Emacs Lisp or Rust using it here in this package.

Error reporting for ts-make-query is confusing

When a query is invalid, ts-make-query raises an error like this:

(rust-panic "called `Result::unwrap()` on an `Err` value: Syntax(1, \"[(call_expression function: (identifier) @function)]\\n^\")")

This is because _make_query uses unwrap.

It should instead define a new error class, e.g. ts-query-error, and raise errors with good messages.

Write a guide doc

Tree-sitter itself has good documentation, but its target audience is different. The guide doc should:

  • Leave out details irrelevant to writing Emacs packages.
  • Explain Emacs-specific nuances.
  • Provide examples in Lisp.

should we update the echo statements in Makefile?

I wonder if we need to do this given that the lisp files have been moved to a directory in the root level. I'm syncing some of my branches with your master branch, so maybe I will figure this out in a few moments.

No support for rustic-mode

I use rustic-mode instead of rust-mode as the major mode for Rust files in emacs and emacs-tree-sitter has no support for it :(

Can you please add support for rustic-mode

tree-sitter-hl-mode slows python-mode a little

Hi, thanks for this great package first of all.

In python-mode, when there are lots of thing to highlight (and maybe with long files like above 200), emacs is slower with tree-sitter-hl-mode. If I disable it, it speeds up. I can notice it when I hold C-n or C-p. Is it normal? I think tree-sitter supposed to be faster than font-lock.

unexpected highlighting for python

tree-sitter-hl-mode cannot correctly hightlight class names starting with underscore.

minimal example:

class Foo:
    pass


class _Foo:
    pass

behavior:
class Foo has tree-sitter-hl-face:constructor face, but class _Foo has default face.

solution:
replace

((identifier) @constructor
 (#match? @constructor "^[A-Z]"))

in ./langs/queries/python/hightlight.scm with

((identifier) @constructor
 (#match? @constructor "^[A-Z_]"))

caveat:
I am not sure whether the current highlighting scheme is intentional (i.e., do not put tree-sitter-hl-face:constructor to private classes). If so, you can close the issue.

thanks!

Intelligent cursor restoration after auto-formatting

I've never used tree-sitter so let me know if this is not possible or doesn't make much sense.

Restoring buffer position after applying styling fixes can be difficult. My frequent use case is with the black formatter using https://github.com/lassik/emacs-format-all-the-code on save.

Black makes a guarantee that the AST is unmodified after formatting, so it seems to me that combining tree-sitter with this the user could be placed on the same tree node as they were before the format. This would make it so that autoformatting would leave your cursor in the same logical place as it was before, even if the number of lines of code was radically changed.

Args out of range error during tree-sitter-langs installation

For a couple of days now I've been getting the following error on a new TS install:

Warning (initialization): An error occurred while loading ‘/home/tim/.emacs.d/init.el’:

Args out of range: #<killed buffer>, 513, 16463596559

This is from Emacs 27.1, started with --debug-init
Config:

(unless (package-installed-p 'tree-sitter)
  (package-install 'tree-sitter)
  (package-install 'tree-sitter-langs))
(require 'tree-sitter)
(require 'tree-sitter-langs)

(First had it with use-package but went with a vanilla config to ensure it's not the setup)

straight.el stuck at building tsc

with recent commits, emacs stuck at building tree-sitter -> building tsc. Old version works fine(without tsc)

  • emacs version: 28 native-comp
  • tree-sitter version: 2fa5045
  • system: x86_64 linux

my config is same as doc:

(straight-register-package
 '(tsc :host github
       :repo "ubolonton/emacs-tree-sitter"
       :files ("core/*.el")))
;; Base framework, syntax highlighting.
(straight-use-package
 '(tree-sitter :host github
               :repo "ubolonton/emacs-tree-sitter"
               :files ("lisp/*.el")))
;; Language bundle.
(straight-use-package
 '(tree-sitter-langs :host github
                     :repo "ubolonton/emacs-tree-sitter"
                     :files ("langs/*.el" "langs/queries")))

Mention how to install tree-sitter-cli in README

When I run ./bin/ensure-lang python, it returns an error

Running 'tree-sitter test' for python
./bin/ensure-lang: line 35: tree-sitter: command not found

And I went out to search how to install tree-sitter's cli.

It would be nice if you mention that tree-sitter-cli is required and how to install it.

Crash when commenting code

I'm trying this out with tree-sitter-bash.

init.el:

(push "~/emacs-tree-sitter" load-path)
(require 'tree-sitter)
(add-hook 'sh-mode-hook (lambda ()
  (setq tree-sitter-language (ts-load-language "bash"))))

In this file:

ls
  1. M-x tree-sitter-mode
  2. Highlight the line
  3. M-x comment-or-uncomment-region
Debugger entered--Lisp error: (args-out-of-range #<buffer a.sh> 1 6)
  ts-parse(#<user-ptr ptr=0x600000009be0 finalizer=0x10dd17c70> ts-buffer-input #<user-ptr ptr=0x60000000a3c0 finalizer=0x10dd17d30>)
  (setq tree-sitter-tree (ts-parse tree-sitter-parser (function ts-buffer-input) tree-sitter-tree))
  (let ((old-tree tree-sitter-tree)) (setq tree-sitter-tree (ts-parse tree-sitter-parser (function ts-buffer-input) tree-sitter-tree)) (run-hook-with-args (quote tree-sitter-after-change-functions) old-tree))
  tree-sitter--do-parse()
  (progn (ts-edit-tree tree-sitter-tree tree-sitter--start-byte tree-sitter--old-end-byte tree-sitter--new-end-byte tree-sitter--start-point tree-sitter--old-end-point tree-sitter--new-end-point) (tree-sitter--do-parse))
  (if tree-sitter-tree (progn (ts-edit-tree tree-sitter-tree tree-sitter--start-byte tree-sitter--old-end-byte tree-sitter--new-end-byte tree-sitter--start-point tree-sitter--old-end-point tree-sitter--new-end-point) (tree-sitter--do-parse)))
  tree-sitter--after-change(1 3 0)
  comment-region-internal(1 3 "# " nil nil nil nil nil t)
  comment-region-default(1 3 nil)
  comment-region(1 3 nil)
  comment-or-uncomment-region(1 3 nil)
  funcall-interactively(comment-or-uncomment-region 1 3 nil)
  call-interactively(comment-or-uncomment-region record nil)
  command-execute(comment-or-uncomment-region record)
  execute-extended-command(nil "comment-or-uncomment-region" "comment-o")
  funcall-interactively(execute-extended-command nil "comment-or-uncomment-region" "comment-o")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

[question] future plans

After implementing highlights what are your plans for the emacs-tree-sitter?

I am particularly interested in implementing operations over the AST in the spirit of smartparens/paredit.

Install grammar binaries relative to user-emacs-directory, instead of using locate-library

Many emacs packages, e.g. lsp-mode, install binaries relative to ~/.emacs.d and allow the binary location to be determined through a variable.

In contrast, this package installs its non-elisp data relative to the compiled library (via load-library). This breaks when using gccemacs, because gccemacs compiles files into their own directory.

If you used the convention and installed into ~/.emacs.d/ , or at least allowed the user to control where the data was installed, it would fix this problem.

Thank you!

Wrong highlighting of field declarations in c/c++

Hi!
In field declarations within structs/classes only the first field in comma separated lists of fields having common type gets assigned correct face (tree-sitter-hl-face:property.definition). The rest fields are assigned only (tree-sitter-hl-face:property). Also see screenshot below. I use f5065c8 with Emacs 27.1.

image

emacs-tree-sitter has issues with auto-yasnippet: wrong-type-argument char-or-string-p nil

After creating a snippet, when I expand the snippet in a Python buffer, I got this error:

Debugger entered--Lisp error: (wrong-type-argument char-or-string-p nil)
  tree-sitter--after-change(2045 2151 106)
  run-hook-with-args(tree-sitter--after-change 2045 2151 106)
  #f(compiled-function () #<bytecode 0x446903a9>)()
  funcall(#f(compiled-function () #<bytecode 0x446903a9>))
  (let nil (funcall (quote #f(compiled-function () #<bytecode 0x446903a9>))))
  eval((let nil (funcall (quote #f(compiled-function () #<bytecode 0x446903a9>)))))
  yas--snippet-create("if data.$1:\n            feed = team_pb2.UpdateTeamInput.$1.DESCRIPTOR.name\n            mask.paths.append(feed)" nil 2045 2045)
  yas-expand-snippet("if data.$1:\n            feed = team_pb2.UpdateTeamInput.$1.DESCRIPTOR.name\n            mask.paths.append(feed)")
  aya-expand()
  funcall-interactively(aya-expand)
  call-interactively(aya-expand nil nil)
  command-execute(aya-expand)

the result is that the string is expanded, but those parts with $1 are empty.

My Emacs version: GNU Emacs 26.3 (build 1, x86_64-apple-darwin18.2.0, NS appkit-1671.20 Version 10.14.3 (Build 18D109)) of 2019-09-02

Conflict with evil-multiedit

Hi, I am using your amazing package from Doom Emacs and I found a conflict between tree-sitter-hl-mode and evil-multiedit.

To reproduce:

  • Select a few occurrences of a symbol with evil-multiedit-match-symbol-and-next, one occurrence in a string and one occurrence in a variable name
  • Try to edit them at once (for instance append some characters at the end) and it will not work.

I work with python and both packages are essential to my workflow! Please let me know if I can do anything to help you debug!

Two tree-sitter.el implementations

@ubolonton and @karlotness are you aware of each other's tree-sitter.el implementations?

This is going to be a problem once you both want to get your implementations added to GNU Elpa or Melpa.

I haven't a concrete proposal, but suggest that the two of you talk about this. Oh, and you might want to mention the other implementation in your README's and if you already had such a talk, then also link to that.

Cycling among different highlight files?

TLDR:

I've been working on highlights.scm files for tree-sitter-clojure and have begun to think that being able to cycle through some different highlighting schemes might be quite useful (please see below for background of the idea).

I scanned the current emacs lisp files a bit and the impression I got was that the idea is to have basically have one type of highlighting (possibly with customizations?). Does that seem correct?

If the idea of cycling through some different highlighting schemes sounds palatable, any suggestions / pointers on implementing such functionality?

Thanks for the consideration.

Background:

I watched a talk (https://youtu.be/l1b7Da2DnPo?t=831) by @tonsky recently where he discusses his alabaster theme: https://github.com/tonsky/sublime-scheme-alabaster

I implemented an approximation to it, gave it a try and found that it might be helpful for me sometimes (e.g. primarily when first examining unfamiliar code), but I doubt I would find it useful all of the time.

To give a few examples, it makes comments stand out, which I find helpful sometimes but not usually. It also does not highlight "built-in" functions / macros / special forms. When I am not yet familiar with a particular language, I would prefer to have such things highlighted and if possible be able to tell built-ins apart from "user-defined". Even if I am familiar, I thought I might be helped to be able to temporarily turn on such highlighting.

FWIW, the aforementioned repository has some details on the rationale behind some of his decisions and I found this to be interesting reading. I don't happen to agree with multiple issues, but I would guess that whatever I were to express as a list of important points, someone else would likely have their own differing opinion.

These experiences suggest to me that perhaps it would be useful to have a few different highlighting schemes one can switch among for different purposes. Perhaps one can think of them as different "glasses" for different occasions.

The query syntax seems to be evolving to make more things possible (e.g. possibly some form of not quantifier may show up at some point: tree-sitter/tree-sitter#705) and my experience with it so far suggests it may be easier to put together (as well as modify) different schemes than some previous valiant efforts (https://github.com/clojure-emacs/clojure-mode/blob/master/clojure-mode.el#L750-L926).

Integration with imenu

A simplistic imenu-create-index-function can look like this:

(defun tree-sitter-rust-imenu-index-function ()
  (thread-last (tree-sitter-query [(function_item (identifier) @function)])
    (seq-map (lambda (capture)
               (pcase-let ((`(_ . ,node) capture))
                 (cons (ts-node-text node)
                       (ts-node-start-position node)))))))

A more substantial implementation can use ts-query-matches to capture nested structures.

(require 'init-tree-sitter) error

when eval (require 'init-tree-sitter), it throw error below:

Debugger entered--Lisp error: (args-out-of-range #<buffer  *tar-data  *temp**> 513 16463596559)
  write-region(#<marker at 513 in  *tar-data  *temp**> 16463596559 "\037\213\010")
  tar-untar-buffer()
  (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer))
  (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  (catch (quote --cl-block-nil--) (if (string= version current-version) (if skip-if-installed (progn (message "tree-sitter-langs: Grammar bundle v%s was already installed; skipped" version) (throw (quote --cl-block-nil--) nil)) (message "tree-sitter-langs: Grammar bundle v%s was already installed; reinstalling" version)) (message "tree-sitter-langs: Installing grammar bundle v%s (was v%s)" version current-version)) (url-copy-file (tree-sitter-langs--bundle-url version os) bundle-file (quote ok-if-already-exists)) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))) (let ((temp-file tree-sitter-langs--bundle-version-file) (temp-buffer (get-buffer-create (generate-new-buffer-name " *temp file*")))) (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (let ((coding-system-for-write ...)) (insert version))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))) (if keep-bundle nil (delete-file bundle-file (quote trash))) (if (and (called-interactively-p (quote any)) (y-or-n-p (format "Show installed grammars in %s? " tree-sitter-langs--bin-dir))) (progn (save-current-buffer (set-buffer (find-file tree-sitter-langs--bin-dir)) (if (and (boundp (quote dired-omit-mode)) dired-omit-mode) (progn (dired-omit-mode -1)))))))
  (let* ((version (or version tree-sitter-langs--bundle-version)) (default-directory tree-sitter-langs--bin-dir) (bundle-file (tree-sitter-langs--bundle-file ".gz" version os)) (current-version (if (file-exists-p tree-sitter-langs--bundle-version-file) (progn (let ((temp-buffer ...)) (save-current-buffer (set-buffer temp-buffer) (unwind-protect ... ...))))))) (catch (quote --cl-block-nil--) (if (string= version current-version) (if skip-if-installed (progn (message "tree-sitter-langs: Grammar bundle v%s was already installed; skipped" version) (throw (quote --cl-block-nil--) nil)) (message "tree-sitter-langs: Grammar bundle v%s was already installed; reinstalling" version)) (message "tree-sitter-langs: Installing grammar bundle v%s (was v%s)" version current-version)) (url-copy-file (tree-sitter-langs--bundle-url version os) bundle-file (quote ok-if-already-exists)) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents bundle-file) (tar-mode) (tar-untar-buffer)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))) (let ((temp-file tree-sitter-langs--bundle-version-file) (temp-buffer (get-buffer-create (generate-new-buffer-name " *temp file*")))) (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (let (...) (insert version))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))) (if keep-bundle nil (delete-file bundle-file (quote trash))) (if (and (called-interactively-p (quote any)) (y-or-n-p (format "Show installed grammars in %s? " tree-sitter-langs--bin-dir))) (progn (save-current-buffer (set-buffer (find-file tree-sitter-langs--bin-dir)) (if (and (boundp ...) dired-omit-mode) (progn (dired-omit-mode -1))))))))
  tree-sitter-langs-install-grammars(:skip-if-installed)
  (if (bound-and-true-p tree-sitter-langs--testing) nil (tree-sitter-langs-install-grammars :skip-if-installed))
  (unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed))
  (progn (unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed)))
  eval((progn (unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed))) t)
  #f(compiled-function (&rest body) "Like `progn', but evaluates the body at compile time if you're compiling.\nThus, the result of the body appears to the compiler as a quoted\nconstant.  In interpreted code, this is entirely equivalent to\n`progn', except that the value of the expression may be (but is\nnot necessarily) computed at load time if eager macro expansion\nis enabled." #<bytecode 0x1a73d7>)((unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed)))
  (eval-when-compile (unless (bound-and-true-p tree-sitter-langs--testing) (tree-sitter-langs-install-grammars :skip-if-installed)))
  eval-buffer(#<buffer  *load*-564722> nil "/home/kai/.emacs.d/lib/tree-sitter/langs/tree-sitter-langs.el" nil t)  ; Reading at buffer position 1671
  load-with-code-conversion("/home/kai/.emacs.d/lib/tree-sitter/langs/tree-sitter-langs.el" "/home/kai/.emacs.d/lib/tree-sitter/langs/tree-sitter-langs.el" nil t)
  #<subr require>(tree-sitter-langs)
  apply(#<subr require> tree-sitter-langs)
  require(tree-sitter-langs)

Doesn't compile on Rust 1.47

On Rust 1.47, compilation fails with a lot of errors similar to this:

error[E0423]: expected value, found macro `env`
   --> core/src/lang.rs:182:52
    |
182 |     "lang-node-type-named-p" fn node_kind_is_named(type_id: u16) -> bool
    |                                                    ^^^^^^^ not a value
    |
help: use `!` to invoke the macro
    |
182 |     "lang-node-type-named-p" fn node_kind_is_named(type_id!: u16) -> bool
    |                                                           ^
help: consider importing this function instead
    |
1   | use crate::lang::os::unix::process::sys::os::env;
    |

Consider converting the point in the library

ATM the offset has to be 1+ or 1- because tree sitter is zero based while emacs 1 based. IMO it will be better to fix that now in the tree-sitter library until there aren't a lot of clients of emacs tree-sitter bindings.

Support minor modes that rely on font-lock-mode

Currently tree-sitter-hl-mode disables all of font-lock-mode's highlighting.

It should instead disable only patterns set by the major mode, i.e. font-lock-defaults, and respect patterns added by minor modes and end users (through font-lock-add-keywords).

Note: This is a bit tricky, as the way font-lock-mode supports custom patterns is messy. See e.g. font-lock-refresh-defaults.

Define what external dependencies are required

After installing Rust (with Rustup) and running './bin/build` I get this:

❯ ./bin/build
   Compiling emacs_module v0.10.0
error: failed to run custom build command for `emacs_module v0.10.0`

Caused by:
  process didn't exit successfully: `/home/jorge/code/emacs/emacs-tree-sitter/target/debug/build/emacs_module-a6842dec63007bb2/build-script-build` (exit code: 101)
--- stderr
./include/emacs-module.h:24:10: fatal error: 'stddef.h' file not found
./include/emacs-module.h:24:10: fatal error: 'stddef.h' file not found, err: true
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ()', src/libcore/result.rs:1084:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

It seems I don't have all dependencies on my Linux Mint 19.x (work laptop; I use Arch Linux at home and I'm more familiar with it), knowing what extra packages to install would be of use. I got Emacs from Snap

I have build-essential installed.

syntax highlighting for c++ removes font-lock-variable-name-face

currently syntax highlighting in c++ is too much. When everything is highlighted on the page nothing becomes discernible. Currently I need to set tree-sitter-hl-face:variable to inherit from default just to regain some sanity. But then I lose variable name highlighting (during a variables declaration that is). How can I retain some features from the default font lock (such as font-lock-variable-name-face) without turning off tree-sitter?

Need help with C++ function indexing query

Experimenting with a drop in replacement for imenu indexing, per the other issue mentioning it. I'm not having much success getting a query capture back from the tools, and was hoping for some help.

Here's the query and command:

(defvar joe:func-query
  (ts-make-query (tree-sitter-require 'cpp) [(function_definition (identifier) @function)]))

(defun joe:make-index (capture)
  (pcase-let ((`(_ . ,node) capture))
    (cons (ts-node-text node)
      (ts-node-start-position node))))

(defun tree-sitter-c++-imenu-index-function ()
  (thread-last
    (ts-root-node tree-sitter-tree)
    (ts-query-captures joe:func-query)
    (seq-map #'joe:make-index)))

Running tree-sitter-debug-enable in the C++ file generates a full top-down tree, with plenty of function_definition nodes.

If I run a call to eval expression with the contents
(ts-query-captures joe:func-query (ts-root-node tree-sitter-tree))
it comes back with an empty vector, "[]"

Suggestions for debugging?

Potential namespace conflict with ts.el

Hi @ubolonton,

Following the recent discussions on emacs-devel, I saw you mention a ts-parse function. I can't find that function's definition in the master branch, but I do see some calls to it. So I thought I should mention that there is a potential namespace conflict with my library https://github.com/alphapapa/ts.el, which implements a ts-parse function.

Thanks for your work on this package. I'm looking forward to the improvements it will bring to Emacs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recently View Projects