samvv / mage Goto Github PK
View Code? Open in Web Editor NEWA work-in-progress lexer/parser generator for a growing number of languages
A work-in-progress lexer/parser generator for a growing number of languages
Idea taken from the tee-sitter documentation.
Something like the following:
@keyword
pub token ident = [a-z_]+
Assume the following code is generated:
Stmt = RetStmt | ExprStmt
This expression:
if isisinstance(field_0, Stmt):
...
is not valid and should be replaced by a call to a dedicated predicate.
For instance, when parsing an integer that only consists of zeros (like 000000
), we could use the number 6 to represent the entire sequence.
This will require adjusting the logic in gen_init_body
to reflect its older form when at least one element is required.
Take this Python snippet:
def foo(a: int, b: str, c: bool) -> None:
print("Hello")
Something that is equally valid:
def foo(
a: int,
b: str,
c: bool
) -> None:
print("Hello")
In the second example, the parameters are wrapped and take up less horizontal space.
We'd like that Mage generator to handle this case automatically for any given grammar.
This might require generating a pseudo-emit function that just calculates the horizontal depth of the block being generated.
See this answer on StackExchange for more information on backtracking.
The following expression:
pub foo = 'hello' \[\s]+ 'world
Isn't valid because the emitter cannot reconstruct at least one \s
out of the CST that was generated.
The following would be valid:
pub foo = 'hello' \[\s]* 'world'
In this case the emitter would just output zero characters.
From the tree-sitter documentation:
extras - an array of tokens that may appear anywhere in the language. This is often used for whitespace and comments. The default value of extras is to accept whitespace. To control whitespace explicitly, specify extras: $ => [] in your grammar.
Something like the following:
@skip
token skip = [\s]*
Right now, such a token holds plain text, but there might be better ways to represent the different choices.
Somewhat related to issue #3.
A CST should be a pure textual representation of the underlying source. This means that types like int
and float
should not be used to represent parts of the CST.
Let's say we have the following grammar:
@skip
skip = [\s]*
pub token expr1 = [a-z]+
pub token expr2 = 'bar'
pub foo = expr1 expr2
expr1
is defined in such a way that it completely consumes expr2
.
During transformation, foo
is injected with the skip
rule:
pub foo = expr1 skip expr2
We would like the code emitter to force emit a whitespace between expr1
and expr2
.
Why should it do it?
skip
allows for such an emitted string to be parsed.expr2
, such as ba
, is consumed by expr1
during a re-parse if it does not emit a whitespaceThis boils down to adding an additional step in the code emitter where every skip
in the grammar is checked and converted to some additional statements for emitting the whitespace. It is also possible to do part of this at runtime, where the whitespace is only emitted if it is strictly necessary.
The following is a snippet of some generated code that is missing any checks:
new_params = list()
for params_element in params:
params_element_0 = params_element[0]
new_params_element_0 = params_element_0
See this answer, this answer and also this answer for more information.
This is part of issue #15.
The default type for these nodes can be something like a ByteString
, which just takes the literal text of the token.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.