Suggestion for keeping some terminals about lark

[richardcooper]

I'll also note that !?rule works but switching the order to ?!rule is not allowed.

[erezsh]

Hi Richard, one simple way to keep a token in the tree, is to give it a name. For example:

?type: "(" type ")"
     | BOOL
     | type "→" type

BOOL: "bool"

However, a little design suggestion: It's usually better to name branches (for example, using an alias) rather than leave unnecessary tokens. It makes the tree easier to process later, when using transformers, and imho just in general.

[richardcooper]

one simple way to keep a token in the tree, is to give it a name

Yeah, I considered that too. I ended up needing to name almost all of the my tokens which was a bit heavyweight for me. The !rule approach works better for my particular needs. I just thought I'd suggest having a lightweight way to enable token keeping on a per-token basis rather than per-rule (!rule) or per-grammar (keep_all_tokens ). If you don't see the need then I can live with using !rule.

It's usually better to name branches (for example, using an alias) rather than leave unnecessary tokens.

I totally agree, if what you are doing is writing a parser for a known language. Which probably 99% of the time is what users of lark will be doing.

I'm doing something a bit different though, I'm taking user-supplied EBNF, using that to parse user-supplied strings, Doing Stuff with the resulting parse tree, and then displaying it. So because I don't actually know the language being parsed, if I throw away tokens I have no way to recover and render them later.

[erezsh]

I'm not entirely against adding a modifier to anonymous tokens (perhaps: "token"! )
I'll be happy to hear more about this use-case. Maybe it will persuade me that such a modifier is useful, or maybe I can suggest a better solution. Also, I'm curious about the purpose of your project. (email is okay too)

[kyouko-taiga]

One use-case I can think of is a less invasive solution to determine the location of a subtree (see #12). Let's say I have a rule like that:

fun_decl: "fun" NAME "(" param_list? ")" "->" type_ident "{" stmt_list? "}"

It might be desirable to access the first and last terminals (resp. "fun" and "}") to determine the start and end locations of a function declaration.

[erezsh]

If Tree nodes (i.e. rules) had a 'range' attribute, indicating the start and end position of the rule in the text, would that solve the problem?

Nevermind, let's discuss this in issue #12

[erezsh]

@richardcooper Did you manage to make it work with keep_all_tokens?
If not, I have a suggestion for you, if you don't mind getting a little creative.

[richardcooper]

keep_all_tokens isn't fully implemented yet is it? I'm currently using !rules like this:

    !?type: bracketed_type
          | "Bool"
          | type "→" type

    ?bracketed_type: "(" type ")"

in order to keep the "→" but discard the brackets.

I'm open to more creative solutions.

[erezsh]

Sorry, my mistake, I just now pushed the patch to master. keep_all_tokens should work now.

I'll get back to you regarding the creative solution, it still needs some working out.

[erezsh]

Closing this issue. Feel free to open a new one if you need anything else.