tree-sitter / tree-sitter-rust

Rust grammar for tree-sitter

tree-sitter-rust

Rust grammar for tree-sitter.

Features

• Speed — When initially parsing a file, tree-sitter-rust takes around two to three times as long as rustc's hand-written parser.

  $ wc -l examples/ast.rs
    2157 examples/ast.rs
  
  $ rustc -Z unpretty=ast-tree -Z time-passes examples/ast.rs | head -n0
    time:   0.002; rss:   55MB ->   60MB (   +5MB)  parse_crate
  
  $ tree-sitter parse examples/ast.rs --quiet --time
    examples/ast.rs    6.48 ms        9908 bytes/ms

  But if you edit the file after parsing it, tree-sitter can generally update the previous existing syntax tree to reflect your edit in less than a millisecond, thanks to its incremental parsing system.

References

• The Rust Reference — While Rust does not have a specification, the reference tries to describe its working in detail. It tends to be out of date.
• Keywords and Operators and Symbols.

tree-sitter-rust's Issues

Raw identifiers are not supported

The following code is valid rust:

static r#message: &str = "Hello, world!";

fn main() {
  println!("{}", message);
}

But it is produces on error in tree-sitter-rust, for example when you enter it in the playground.

Parse errors with the precedence of range literals

I tried to run this code with tree-sitter-rust v0.19.0 and tree-sitter v0.19.4

for i in 0.. {
    if *wide.offset(i) == 0 {
        return Self::from_wide_ptr(wide, i as usize)
    }
}

and I got the following AST:

(source_file [0, 0] - [5, 0]
  (ERROR [0, 0] - [4, 1]
    (identifier [0, 4] - [0, 5])
    (range_expression [0, 9] - [4, 1]
      (integer_literal [0, 9] - [0, 10])
      (block [0, 13] - [4, 1]
        (if_expression [1, 4] - [3, 5]
          condition: (binary_expression [1, 7] - [1, 27]
            left: (unary_expression [1, 7] - [1, 22]
              (call_expression [1, 8] - [1, 22]
                function: (field_expression [1, 8] - [1, 19]
                  value: (identifier [1, 8] - [1, 12])
                  field: (field_identifier [1, 13] - [1, 19]))
                arguments: (arguments [1, 19] - [1, 22]
                  (identifier [1, 20] - [1, 21]))))
            right: (integer_literal [1, 26] - [1, 27]))
          consequence: (block [1, 28] - [3, 5]
            (return_expression [2, 8] - [2, 52]
              (call_expression [2, 15] - [2, 52]
                function: (scoped_identifier [2, 15] - [2, 34]
                  path: (identifier [2, 15] - [2, 19])
                  name: (identifier [2, 21] - [2, 34]))
                arguments: (arguments [2, 34] - [2, 52]
                  (identifier [2, 35] - [2, 39])
                  (type_cast_expression [2, 41] - [2, 51]
                    value: (identifier [2, 41] - [2, 42])
                    type: (primitive_type [2, 46] - [2, 51])))))))))))

Hard to get some symbols inside macro calls and definitions on the user side

Not sure if it's a proper bug, but I got myself unable to classify some symbols on Atom inside macros.

Like, when I define something like:

'"+"' : 'keyword.operator.add'

It will classify all + signs, except inside macro invocations and macro definitions

The same are true for all operator-like symbols and the in, but other keywords work just fine.

Looking at the code, I suspect that this happens inside _non_special_token, where in is not there and the operator-like symbols are defined by a regular expression /[/_\-=->,;:::!=?.@*=/='&=#%=^=+<>|~]+/, not separated strings.

Again, not sure if it can be considered a bug or if it should be handled in some way on the user side.

Parse error in syntactically-valid program

Code extracted from rustc souce code:

fn hash_body(
    item_like: impl for<'a> HashStable<StableHashingContext<'a>>,
    hir_body_nodes: &mut Vec<(DefPathHash, Fingerprint)>,
) -> Fingerprint {
    let hash = hash(hcx, HirItemLike { item_like: &item_like });
}

fn upstream_crates() {
    let mut upstream_crates: Vec<_> = cstore;
}

This program is syntactically valid, but using the master branch of tree-sitter-rust currently this can't be parsed. Current AST:

(ERROR [0, 0] - [10, 0]
  (identifier [0, 3] - [0, 12])
  (identifier [1, 4] - [1, 13])
  (for_lifetimes [1, 20] - [1, 27]
    (lifetime [1, 24] - [1, 26]
      (identifier [1, 25] - [1, 26])))
  (ERROR [1, 28] - [4, 14]
    (generic_type [1, 28] - [1, 64]
      type: (type_identifier [1, 28] - [1, 38])
      type_arguments: (type_arguments [1, 38] - [1, 64]
        (generic_type [1, 39] - [1, 63]
          type: (type_identifier [1, 39] - [1, 59])
          type_arguments: (type_arguments [1, 59] - [1, 63]
            (lifetime [1, 60] - [1, 62]
              (identifier [1, 61] - [1, 62]))))))
    (identifier [2, 4] - [2, 18])
    (mutable_specifier [2, 21] - [2, 24])
    (generic_type [2, 25] - [2, 56]
      type: (type_identifier [2, 25] - [2, 28])
      type_arguments: (type_arguments [2, 28] - [2, 56]
        (tuple_type [2, 29] - [2, 55]
          (type_identifier [2, 30] - [2, 41])
          (type_identifier [2, 43] - [2, 54]))))
    (identifier [3, 5] - [3, 16])
    (identifier [4, 8] - [4, 12]))
  (identifier [4, 15] - [4, 19])
  (type_identifier [4, 20] - [4, 23])
  (ERROR [4, 25] - [7, 2]
    (struct_pattern [4, 25] - [4, 62]
      type: (type_identifier [4, 25] - [4, 36])
      (field_pattern [4, 39] - [4, 60]
        name: (field_identifier [4, 39] - [4, 48])
        pattern: (reference_pattern [4, 50] - [4, 60]
          (identifier [4, 51] - [4, 60])))))
  (identifier [7, 3] - [7, 18])
  (ERROR [7, 18] - [7, 20]
    (parameters [7, 18] - [7, 20]))
  (ERROR [8, 4] - [8, 7]
    (identifier [8, 4] - [8, 7]))
  (mutable_specifier [8, 8] - [8, 11])
  (identifier [8, 12] - [8, 27])
  (generic_type [8, 29] - [8, 35]
    type: (type_identifier [8, 29] - [8, 32])
    type_arguments: (type_arguments [8, 32] - [8, 35]
      (type_identifier [8, 33] - [8, 34])))
  (identifier [8, 38] - [8, 44]))

It seems like the problem is the type impl for<'a> HashStable<StableHashingContext<'a>>.

Doc strings parser

It would be a nice enhancement if doc string comments would be parsed, to provide better highlights when documenting code

default is no more a function modifier so remove it

According to https://doc.rust-lang.org/reference/items/functions.html we could remove default.
Its presence in the grammar leads to error with for example:

pub struct A {
    a: default::C,
}

And nothing in:
https://doc.rust-lang.org/reference/keywords.html

@maxbrunsfeld are you ok to remove it ? if yes I'll do the PR.

[BUG] Error when attribute inside struct declaration

Looks like is the lack of possibility to have a attribute item inside struct_expression

Example:

struct MyType {
    aaaaa: bool,
}

impl MyType {
    fn new() -> Self {
        MyType {
            #[cfg(aaaa)]
            aaaaa,
        }
    }
}

Result:

(source_file [1, 0] - [13, 0]
  (struct_item [1, 0] - [3, 1]
    (type_identifier [1, 7] - [1, 13])
    (field_declaration_list [1, 14] - [3, 1]
      (field_declaration [2, 4] - [2, 15]
        (field_identifier [2, 4] - [2, 9])
        (primitive_type [2, 11] - [2, 15]))))
  (impl_item [5, 0] - [12, 1]
    (type_identifier [5, 5] - [5, 11])
    (declaration_list [5, 12] - [12, 1]
      (function_item [6, 4] - [11, 5]
        (identifier [6, 7] - [6, 10])
        (parameters [6, 10] - [6, 12])
        (type_identifier [6, 16] - [6, 20])
        (block [6, 21] - [11, 5]
          (struct_expression [7, 8] - [10, 9]
            (type_identifier [7, 8] - [7, 14])
            (field_initializer_list [7, 15] - [10, 9]
              (ERROR [8, 12] - [9, 17]
                (attribute_item [8, 12] - [8, 24]
                  (meta_item [8, 14] - [8, 23]
                    (identifier [8, 14] - [8, 17])
                    (meta_item [8, 18] - [8, 22]
                      (identifier [8, 18] - [8, 22]))))
                (identifier [9, 12] - [9, 17])))))))))
test.rs	0 ms	(ERROR [8, 12] - [9, 17])

Highlight `box`

Now that the box keyword has been reserved in Rust2018, and it is possible to use it behind a gate I think it would make sense to highlight it as a keyword

Operators are not being highlighted

Other tree-sitter grammars parse ->, &&, +=, etc. as an operator.
This grammar should be consistent with the standard convention.

Mismatch between WASM and native builds

There seems to be a mismatch between WASM and native builds (on macOS).

I built the CLI from latest tree-sitter's master (4c0fa29) and tried this code:

macro_rules! impl_pred {}

// TODO
i
impl_pred!(foo, bar);

This is the syntax tree reported by the native binding (tree-sitter test passes for this commit):

(source_file
 (macro_definition name:
                   (identifier))
 (line_comment)
 (macro_invocation macro:
                   (identifier)
                   (ERROR
                    (identifier))
                   (token_tree
                    (identifier)
                    (identifier))))

This is the syntax tree reported by WASM (through tree-sitter web-ui):

(source_file
 (macro_definition name:
                   (identifier))
 (line_comment)
 (identifier)
 (MISSING ";")
 (macro_invocation macro:
                   (identifier)
                   (token_tree
                    (identifier)
                    (identifier))))

Indented comments don't get continued lines indented

Describe the bug

Indented comments don't wrap onto indented new lines in rust. Seems to be ok in the lua plugin.

To Reproduce

Steps to reproduce the behavior:

1. nvim a.rs

struct A;

impl A {
    /// Do some stuff!! (put cursor here and press enter)
    fn a();
}

3. Get

struct A;

impl A {
    /// Do some stuff!!
/// |<- cursor here
    fn a();
}

Expected behavior

struct A;

impl A {
    /// Do some stuff!!
    /// |<- cursor here
    fn a();
}

Output of :checkhealth nvim_treesitter

health#nvim_treesitter#check

Installation

• OK: tree-sitter found 0.19.4 (parser generator, only needed for :TSInstallFromGrammar)
• OK: node found v16.1.0 (only needed for :TSInstallFromGrammar)
• OK: git executable found.
• OK: cc executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl" }
• OK: Neovim was compiled with tree-sitter runtime ABI version 13 (required >=13). Parsers must be compatible with runtime ABI.

Parser/Features H L F I

• c_sharp ✓ . . .
• svelte ✓ . ✓ ✓
• beancount ✓ . ✓ .
• zig ✓ ✓ ✓ ✓
• c ✓ ✓ ✓ ✓
• teal ✓ ✓ ✓ ✓
• java ✓ ✓ . ✓
• python ✓ ✓ ✓ ✓
• bibtex ✓ . ✓ ✓
• sparql ✓ ✓ ✓ ✓
• comment ✓ . . .
• gomod ✓ . . .
• dart ✓ ✓ . ✓
• lua ✓ ✓ ✓ ✓
• latex ✓ . ✓ .
• ocaml ✓ ✓ ✓ .
• bash ✓ ✓ ✓ .
• r ✓ ✓ . .
• yaml ✓ ✓ ✓ ✓
• json ✓ ✓ ✓ ✓
• jsdoc ✓ . . .
• ledger ✓ . ✓ ✓
• php ✓ ✓ ✓ ✓
• julia ✓ ✓ . .
• supercollider ✓ ✓ ✓ ✓
• devicetree ✓ ✓ ✓ ✓
• gdscript ✓ ✓ . .
• turtle ✓ ✓ ✓ ✓
• vue ✓ . ✓ .
• html ✓ ✓ ✓ ✓
• typescript ✓ ✓ ✓ ✓
• regex ✓ . . .
• verilog ✓ ✓ ✓ .
• jsonc ✓ ✓ ✓ ✓
• ql ✓ ✓ . ✓
• fennel ✓ ✓ . .
• rst ✓ ✓ . .
• nix ✓ ✓ ✓ .
• query ✓ ✓ ✓ ✓
• graphql ✓ . . ✓
• cpp ✓ ✓ ✓ ✓
• toml ✓ ✓ ✓ ✓
• kotlin ✓ . . .
• ruby ✓ ✓ ✓ ✓
• fish ✓ ✓ ✓ ✓
• ocamllex ✓ . . .
• dockerfile ✓ . . .
• rust ✓ ✓ ✓ ✓
• ocaml_interface✓ ✓ ✓ .
• javascript ✓ ✓ ✓ ✓
• css ✓ . ✓ ✓
• glimmer ✓ . . .
• erlang . . . .
• clojure ✓ ✓ . .
• go ✓ ✓ ✓ ✓
• tsx ✓ ✓ ✓ ✓

Legend: H[ighlight], L[ocals], F[olds], I[ndents]
+) multiple parsers found, only one will be used
x) errors found in the query, try to run :TSUpdate {lang}

Output of nvim --version

NVIM v0.5.0-dev+1318-g61aefaf29
Build type: RelWithDebInfo
LuaJIT 2.0.5
Compilation: /usr/bin/cc -D_FORTIFY_SOURCE=2 -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -g -fvar-tracking-assignments -fdebug-prefix-map=/home/jade/.cache/aurutils/sync/neovim-git/src=/usr/src/debug -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -O2 -g -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/home/jade/.cache/aurutils/sync/neovim-git/src/build/config -I/home/jade/.cache/aurutils/sync/neovim-git/src/neovim-git/src -I/usr/include -I/home/jade/.cache/aurutils/sync/neovim-git/src/build/src/nvim/auto -I/home/jade/.cache/aurutils/sync/neovim-git/src/build/include
Compiled by jade@tail-bot

Features: +acl +iconv +tui
See ":help feature-compile"

system vimrc file: "$VIM/sysinit.vim"
fall-back for $VIM: "/usr/share/nvim"

Run :checkhealth for more info

Additional context

Treesitter config is:

lua <<EOF
require'nvim-treesitter.configs'.setup {
  ensure_installed = "maintained", -- one of "all", "maintained" (parsers with maintainers), or a list of languages
  -- ignore_install = { "javascript" }, -- List of parsers to ignore installing
  highlight = {
    enable = true,              -- false will disable the whole extension
    --disable = { },  -- list of language that will be disabled
  },
  indent = {
    enable = true
  },
}
EOF

:set indentexpr?: indentexpr=nvim_treesitter#indent()

Incorrect syntax highlighting: the color using the glob-like brace syntax

Expected behavior: the color of the name must be same for any kind of use

Versions

atom --version                                                                                                                                                        master
Atom    : 1.37.0
Electron: 2.0.18
Chrome  : 61.0.316

Unique highlight namess for enum variants and constants

Currently enum variants and constants are considered Constructor, but it would be great if they could be considered something else, as constructor isnt exactly accurate; enums are more like static fields in other languages

Rust syntax highlighting: Destructured slice function arguments

Description

Rust syntax highlighting of destructured slices in function arguments is incorrectly colouring values as types.

Moved here from atom/atom#18571

Steps to Reproduce

Enter code:

fn foo_1([b1]: [u8; 1]) {}
fn foo_2([b1, b2]: [u8; 2]) {}
fn foo_3([b1, b2, b3]: [u8; 3]) {}
fn foo_4([b1, b2, b3, b4]: [u8; 4]) {}
fn foo_5([b1, b2, b3, b4, b5]: [u8; 5]) {}

fn main() {
    let bytes: [u8; 2] = [5, 6];
    let [fx, fy] = bytes;
    let bytes: [u8; 4] = [1, 2, 3, 4];
    let [a, b, c, d] = bytes;
}

Expected behavior: Destructured function arguments slice values should have the same colour as destructured values in the main method.

Actual behavior: Some are coloured like types.

Versions

$ atom --version                                                                                                                                                                                                                                                                       
Atom    : 1.33.0
Electron: 2.0.11
Chrome  : 61.0.3163.100
Node    : 8.9.3

Incorrect Highlighting for tuple parameter names

Case "..=" in range_expression is not defined

One option that exists in Rust for ranges, and the default for normal/non-pattern-matching inclusive range is "..=" and it is missing in the grammar.json

Error on impl block when negating a trait

Example:

impl !Send for Args {}

(source_file
  (impl_item
    (ERROR
      (empty_type))
    trait: (type_identifier)
    type: (type_identifier)
    body: (declaration_list)))

Any unicode character as identifiers (except emojis)

Since this already supports greek characters, and that requires the feature gate
#![feature(non_ascii_idents)]
This package may as well support any unicode character as identifiers, as this feature gate allows to

Parse new macro declarative expression

As reference:

1. rust-lang/rust#39412
2. rust-lang/rfcs#1584
3. https://github.com/rust-lang/rfcs/blob/master/text/1584-macros.md

There is something wrong on with repetition patterns on macros

This not necessary leads to an error, but sometimes do

macro_rules! mything {
    ($($e:expr),+) => {
        $(
            $e
        )* // <- this '*' doesn't get parsed as a string part of the repetition
    };
}

macro_rules! mything {
    ($($e:expr),+) => {
        $(
            $e
        ),* // <- having a item before repetitions pattern is acceptable in rust, but gets messy by tree-sitter-rust
    };
}

That's by no means complete, still testing thing out

Error out when const generics happens

pub fn uninit_array<const LEN: usize>() -> [Self; LEN] {}

This code generate error in the part of const generics.

Error

(source_file [0, 0] - [1, 0]
  (function_item [0, 0] - [0, 57]
    (visibility_modifier [0, 0] - [0, 3])
    name: (identifier [0, 7] - [0, 19])
    type_parameters: (type_parameters [0, 19] - [0, 37]
      (constrained_type_parameter [0, 20] - [0, 36]
        left: (type_identifier [0, 20] - [0, 25])
        (ERROR [0, 26] - [0, 29]
          (identifier [0, 26] - [0, 29]))
        bounds: (trait_bounds [0, 29] - [0, 36]
          (primitive_type [0, 31] - [0, 36]))))
    parameters: (parameters [0, 37] - [0, 39])
    return_type: (array_type [0, 43] - [0, 54]
      element: (type_identifier [0, 44] - [0, 48])
      length: (identifier [0, 50] - [0, 53]))
    body: (block [0, 55] - [0, 57])))

Expected: not sure in terms of nodes that should happen, but not a error

Grammar selector

Grammar selector not showing the Rust as option when in a plain text file file.

Code Folding is Incorrect

Video is here.

Code is here

AST error despite code matching grammar(?)

if let .. = A {  , } {  }

parses with an error as

(source_file (if_let_expression (remaining_field_pattern) (identifier) (block (ERROR))) (block))

but the code appears to match this (error-free) AST:

(source_file (if_let_expression (remaining_field_pattern) (struct_expression (type_identifier) (field_initializer_list)) (block)))

that is, the clause A { , } is an empty struct initialization that is being destructured into .. and only the next pair of curly braces is the if let block.

Not sure what exactly is going on here. Found with tree-saw.

Array and tuple expressions accept inner attributes, not regular attributes

In array_expression and tuple_expression, I'm pretty sure the attributes after the opening bracket should be inner_attribute_item, not attribute_item. See https://doc.rust-lang.org/reference/expressions/array-expr.html . There's a test in declarations.txt that expects the current behavior, and should also be adjusted.

Error when use_wildcard inside of use_list

Example:

Minimal:

use lib::{*};

Result:

(source_file
  (use_declaration
    argument: (scoped_use_list
      path: (identifier)
      list: (use_list
        (ERROR)))))

Expected:

(source_file
  (use_declaration
    argument: (scoped_use_list
      path: (identifier)
      list: (use_list
        (use_wildcard)))))

Usually in a longer form

use lib::{module::Type, *};

Error on New OR patterns

With the new #![feature(or_patterns)]

We get a error on the 2 new syntax it will allow

match a {
    Some(Self::A | Self::B) => "a or b",
    None => "nor A or B"
}

Error:

(source_file [0, 0] - [4, 0]
  (match_expression [0, 0] - [3, 1]
    value: (identifier [0, 6] - [0, 7])
    body: (match_block [0, 8] - [3, 1]
      (match_arm [1, 4] - [1, 40]
        pattern: (match_pattern [1, 4] - [1, 27]
          (tuple_struct_pattern [1, 4] - [1, 27]
            type: (identifier [1, 4] - [1, 8])
            (ERROR [1, 9] - [1, 18]
              (scoped_identifier [1, 9] - [1, 16]
                path: (identifier [1, 9] - [1, 13])
                name: (identifier [1, 15] - [1, 16])))
            (scoped_identifier [1, 19] - [1, 26]
              path: (identifier [1, 19] - [1, 23])
              name: (identifier [1, 25] - [1, 26]))))
        value: (string_literal [1, 31] - [1, 39]))
      (match_arm [2, 4] - [2, 24]
        pattern: (match_pattern [2, 4] - [2, 8]
          (identifier [2, 4] - [2, 8]))
        value: (string_literal [2, 12] - [2, 24])))))
test.rs	0 ms	(ERROR [1, 9] - [1, 18])

let Ok(x) | Err(x) = foo();

Error:

(source_file [0, 0] - [1, 0]
  (let_declaration [0, 0] - [0, 27]
    (ERROR [0, 4] - [0, 11]
      (tuple_struct_pattern [0, 4] - [0, 9]
        type: (identifier [0, 4] - [0, 6])
        (identifier [0, 7] - [0, 8])))
    pattern: (tuple_struct_pattern [0, 12] - [0, 18]
      type: (identifier [0, 12] - [0, 15])
      (identifier [0, 16] - [0, 17]))
    value: (call_expression [0, 21] - [0, 26]
      function: (identifier [0, 21] - [0, 24])
      arguments: (arguments [0, 24] - [0, 26]))))
test.rs	0 ms	(ERROR [0, 4] - [0, 11])

Some erroneous parsing for tree-sitter-rust v0.19

I've used tree-sitter-rust v0.19 to parse the following code:

impl_i!([i8; 2]: i8x2, m8x2 | i8, u8 | test_v16 | x0, x1 |
        From: |
        /// A 16-bit vector with 2 `i8` lanes.
);

AST:

(source_file [0, 0] - [4, 0]
  (macro_invocation [0, 0] - [3, 1]
    macro: (identifier [0, 0] - [0, 6])
    (token_tree [0, 7] - [3, 1]
      (token_tree [0, 8] - [0, 15]
        (primitive_type [0, 9] - [0, 11])
        (integer_literal [0, 13] - [0, 14]))
      (identifier [0, 17] - [0, 21])
      (identifier [0, 23] - [0, 27])
      (primitive_type [0, 30] - [0, 32])
      (primitive_type [0, 34] - [0, 36])
      (identifier [0, 39] - [0, 47])
      (identifier [0, 50] - [0, 52])
      (identifier [0, 54] - [0, 56])
      (identifier [1, 8] - [1, 12])
      (identifier [2, 12] - [2, 13])
      (integer_literal [2, 14] - [2, 16])
      (identifier [2, 17] - [2, 20])
      (identifier [2, 21] - [2, 27])
      (identifier [2, 28] - [2, 32])
      (integer_literal [2, 33] - [2, 34])
      (ERROR [2, 35] - [2, 36]
        (ERROR [2, 35] - [2, 36]))
      (primitive_type [2, 36] - [2, 38])
      (ERROR [2, 38] - [2, 39]
        (ERROR [2, 38] - [2, 39]))
      (identifier [2, 40] - [2, 45]))))

Raw identifiers break the parsing completely

Rust has so called Raw identifiers.

They look like this:

let r#let = 12;
println!("the var let ()", r#let);

This breaks the parsing completely for me.

`as` keyword considered as operator

currently the as keyword is being parsed as an operator, though it is a keyword (as per https://doc.rust-lang.org/reference/keywords.html)

[BUG] Error when use_wildcard inside of scoped_use_list

Minimal example:

use my::{some::*};

Result:

(source_file
  (use_declaration
    (scoped_use_list
      (identifier)
      (use_list
        (identifier)
        (ERROR)))))

Expected:

(source_file
  (use_declaration
    (scoped_use_list
      (identifier)
      (use_list
        (use_wildcard
           (identifier)))))

New issue in const generics

In this code using const generics on enum: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=9ad3aebd75f7e0c3621ffad00165c069

It's possible to have this kind of syntax:

impl Machine<{State::Init}> {}

pub fn accumulate(self) -> Machine<{State::Accumulate}> {}

that gives a error where type_argument would happen

In this other example using &'static str as const generic: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=e957af6f8ba26b8f496366182178a9d2

struct State<const S: &'static str> {}

A error also happen in type_parameter

Parse error on $ in macro invocations

Example:

use syn::Token;
use quote::ToTokens;

fn main() {
    let dollar_sign: Token![$] = Default::default();
    println!("{}", dollar_sign.into_token_stream());
}

This is valid, runnable rust code, but does not parse.

This occurs because the macro_invocation grammar production ultimately delegates to _non_special_token, which excludes $. This is the desired bevarior when parsing macro_rules! declarations, where $ is assigned special meaning, but not when parsing other macro invocations, where $ is just a normal token.

This currently breaks Atom rust syntax highlighting for the scope containing the macro invocation and the rest of the document thereafter.

Support nested block comments

This test should pass:

============================================
Nested block comments
============================================

/*
/* Nested */
*/

----

(source_file
  (block_comment))

Per https://github.com/rust-lang/rust/blob/master/src/test/run-pass/nested-block-comment.rs

`async` and `await` not considered keywords

Support lazy_static `ref` usage

lazy-static is an "as good as std" crate that uses the ref keyword in a way not currently liked by tree-sitter-rust.

I'd expect it to look similar to thread_local! usage.

Atom:

Code:

thread_local! {
    static COSTS: FxHashMap<&'static str, u64> = vec![
        ("foo", 275),
        ("bar", 450),
    ].into_iter().collect();
}

lazy_static! {
    static ref COSTS: FxHashMap<&'static str, u64> = vec![
        ("foo", 275),
        ("bar", 450),
    ].into_iter().collect();
}

Discriminate doc comments from single line comments

It would be helpful to discriminate doc comments (///) from single line comments (//). Currently, they are assigned to the same token

injection query not exposed in the rust crate

Unable to fold 'fn main()' when tree parser is enabled

As the title suggests, I'm not able to fold the main function in Rust when tree parser is enabled. I wasn't sure if this was a bug or something done on purpose, but after checking a few other programming languages under tree parser, such as C++ and others, I see I can fold the main functions in there, so I now assume this is a bug? Not sure.

Should Node be Send and Sync?

In the Rust bindings, I see that Tree is both Send and Sync, but Node isn't. Is that intentional?

non alphanumeric node types

I'm pretty new with rust but I have plenty experience with other languages AST and I'm noticing that, if I iterate descendants directly using children property, for the following code:

pub type __darwin_size_t = ::std::os::raw::c_ulong;

The following ast is generated:

{"type":"source_file","children":[{"type":"type_item","children":[
  {"type":"visibility_modifier","children":
    [{"type":"pub"}]},{"type":"type"},{"type":"type_identifier"},{"type":"="},{"type":"scoped_type_identifier","children":[{"type":"scoped_identifier","children": 
 [{"type":"scoped_identifier","children":[{"type":"scoped_identifier","children":
    [{"type":"::"},{"type":"identifier"}]},{"type":"::"},{"type":"identifier"}]},{"type":"::"},
    {"type":"identifier"}]},{"type":"::"},{"type":"type_identifier"}]},{"type":";"}]}]}

And as you can notice there are node for which their types are are ':', '=', '::', ';'

Is this OK? I read that a convention is to ignore types starting with underscore, so in this case is not clear if I should ignore or is an issue.

IMO these nodes are not ignorable since they add important semantics to the program and I would be loosing information or if not I think without these it will be harder to figure it out.

Again, I'm completely new in rust syntax but in other language they tend to name these kind of node Tokens, like ColonToken. In TypeScript there is even one called "DotDotDotToken" or "OpenTagGreaterThanToken".

Thanks, and sorry if I'm completely wrong about this and is not an issue at all, great project!

Parse error for some valid procedural attribute macros, derive macro helper attributes

tree-sitter-rust parses attributes using the meta-item syntax, which is the syntax used by most of rust's built-in attributes. The meta-item syntax is a subset of the attr syntax. Procedural attribute macros, however, use a larger subset of the attr syntax allowing arbitrary DelimTokenTrees. Derive macro helper attributes use the full attr syntax. As a result, tree-sitter-rust incorrectly parses many valid attributes.

As an example, the following is one of the usage examples for the thiserror crate. thiserror defines a derive macro helper attribute, error:

#[derive(Error, Debug)]
pub enum Error {
    #[error("first letter must be lowercase but was {:?}", first_char(.0))]
    WrongCase(String),
    #[error("invalid index {idx}, expected at least {} and at most {}", .limits.lo, .limits.hi)]
    OutOfBounds { idx: usize, limits: Limits },
} 

The phrases .limits.lo, .limits.hi, and first_char(.0) are legal under the attr syntax but not under meta-item, and so these attributes are parsed incorrectly.

This is likely the cause of atom/atom#20897. Any attribute not conforming to meta-item currently breaks atom's rust syntax highlighting.

Attributes should instead be parsed using the more permissive attr syntax.

Confusion between generic and shift operator

With the expression:

a as *mut A<<B as C>::D>

tree-sitter parses it like

(((a as *mut A) << B) as C) > ::D >

and the final > is erroneous.

I tryed to compile let x = 1 as u32 << 1; and I get:

error: `<` is interpreted as a start of generic arguments for `u32`, not a shift
 --> src/main.rs:2:22
  |
2 |     let x = 1 as u32 << 1;
  |             -------- ^^ - interpreted as generic arguments
  |             |        |
  |             |        not interpreted as shift
  |             help: try shifting the cast value: `(1 as u32)`

I wonder if we could do something similar in the grammar, I mean when we have a rule where a type is expected then a following < of << must be interpret as a part of a generic and not as a bin operator.

Wrong AST due to greedy quantifier in raw_string_literal rule

let a = r#""#; let b = r#""#;

parses as

(source_file (let_declaration (identifier) (raw_string_literal)))

which is wrong. The reason is that the relevant patterns employ greedy quantifiers and thus just skip over the second declaration.

Conceptually, the fix is simply to make the quantifiers lazy instead, but I'm not sure this would actually work since there is also a repeat (why?) and the quotes are not part of the regex itself.

tree-saw now has the ability to predict the AST that tree-sitter "should" produce, which allows it to automatically find issues like this one with no further input or guidance required.

"&" inside a macro isn't recognized

println!("{}", &foo);

Here the query "&" @operator doesn't match the &, if the macro is changed for a function call it works

println("{}", &foo);

Glob imports from enums are incorrectly tagged

When we define an enum, its name has class="syntax--support syntax--type":

enum MyEnum {
    Int(i64),
    Float(f64),
}

When we use the enum, both in return position and in a function body, the name again has class="syntax--support syntax--type":

fn foo() -> MyEnum {
    MyEnum::Int(42)
}

When we import a variant of an enum, the enum name yet again has class="syntax--support syntax--type":

use super::MyEnum::Int;

BUT when we glob import from an enum, the enum name has class="syntax--entity syntax--name syntax--class":

use super::MyEnum::*;

Union syntax must have "{"

Currently tree-sitter-rust allows the following:

union MyUnion;

Where according the reference and the syntax hard error I get from compiler says it must have {
Reference

So a empty union must be

union MyUnion {}

Bad parse of range expression with parenthesized right-side

Hi! I just ran into a parse tree that looks incorrect to me. The following is parsed as a call expression, where the function is itself a range expression - rather than just being parsed as a range expression.

1..(1)

Expected:

 (range_expression (integer_literal) (parenthesized_expression (integer_literal)))

Actual:

(call_expression function: (range_expression (integer_literal)) arguments: (arguments (integer_literal)))

Consecutive `super`s dont get considered as keyword

In the scenario of super::super, only the first super is considered a keyword, whereas the second one isnt considered as anything

Designate "else"/"else if" branches inside of if statements

The same proposal as tree-sitter/tree-sitter-javascript#142 but, this time, for Rust. Quoted:

Currently the else keyword is not captured inside of any node.

I want the whole expression, i.e. including the else keyword, to be scoped inside of a node, so that I can interact fully with it in my editor. Currently, for instance, it's not possible to cut, yank or delete a whole else branch due to this limitation.