samvv / mage Goto Github PK

Idea taken from the tee-sitter documentation.

Something like the following:

@keyword
pub token ident = [a-z_]+

Assume the following code is generated:

Stmt = RetStmt | ExprStmt

This expression:

if isisinstance(field_0, Stmt):
    ...

is not valid and should be replaced by a call to a dedicated predicate.

For instance, when parsing an integer that only consists of zeros (like 000000), we could use the number 6 to represent the entire sequence.

This will require adjusting the logic in gen_init_body to reflect its older form when at least one element is required.

Take this Python snippet:

def foo(a: int, b: str, c: bool) -> None:
    print("Hello")

Something that is equally valid:

def foo(
    a: int,
    b: str,
    c: bool
) -> None:
    print("Hello")

In the second example, the parameters are wrapped and take up less horizontal space.

We'd like that Mage generator to handle this case automatically for any given grammar.

This might require generating a pseudo-emit function that just calculates the horizontal depth of the block being generated.

See this answer on StackExchange for more information on backtracking.

The following expression:

pub foo = 'hello' \[\s]+ 'world

Isn't valid because the emitter cannot reconstruct at least one \s out of the CST that was generated.

The following would be valid:

pub foo = 'hello' \[\s]* 'world'

In this case the emitter would just output zero characters.

From the tree-sitter documentation:

extras - an array of tokens that may appear anywhere in the language. This is often used for whitespace and comments. The default value of extras is to accept whitespace. To control whitespace explicitly, specify extras: $ => [] in your grammar.

Something like the following:

@skip
token skip = [\s]*

Right now, such a token holds plain text, but there might be better ways to represent the different choices.

Somewhat related to issue #3.

A CST should be a pure textual representation of the underlying source. This means that types like int and float should not be used to represent parts of the CST.

Let's say we have the following grammar:

@skip
skip = [\s]*

pub token expr1 = [a-z]+

pub token expr2 = 'bar'

pub foo = expr1 expr2

expr1 is defined in such a way that it completely consumes expr2.

During transformation, foo is injected with the skip rule:

pub foo = expr1 skip expr2

We would like the code emitter to force emit a whitespace between expr1 and expr2.

Why should it do it?

1. It knows it can emit a whitespace because the injected expression skip allows for such an emitted string to be parsed.
2. It knows it must do so because the prefix of expr2, such as ba, is consumed by expr1 during a re-parse if it does not emit a whitespace

This boils down to adding an additional step in the code emitter where every skip in the grammar is checked and converted to some additional statements for emitting the whitespace. It is also possible to do part of this at runtime, where the whitespace is only emitted if it is strictly necessary.

The following is a snippet of some generated code that is missing any checks:

new_params = list()
for params_element in params:
     params_element_0 = params_element[0]
     new_params_element_0 = params_element_0

See this answer, this answer and also this answer for more information.

This is part of issue #15.

The default type for these nodes can be something like a ByteString, which just takes the literal text of the token.