natalie-lang / natalie_parser Goto Github PK

Not sure i've written code like this - I found it out in the wild when testing the parser.

l = 0
while begin l+=1; l<5 end
  puts l
end

08:26 $ ruby -I lib:ext -r natalie_parser -e 'p NatalieParser.parse(File.read("../temp_natalie_parser_tests/whilecond.rb"))'
-e:1:in `parse': (string)#2: syntax error, unexpected name 'l' (expected: 'end-of-line') (SyntaxError)
while begin l+=1; l<5 end
            ^ here, expected 'end-of-line'
        from -e:1:in `<main>'

Our fuzzer found a panic here on #43! 🎉

fuzzing with seed 1106784158
panic: unreachable in /natalie_parser/include/natalie_parser/node/for_node.hpp#39
rake aborted!

Steps to reproduce:

$ gem build
$ gem install natalie_parser-2.3.0.gem
$ irb -r natalie_parser

This works on Ruby 3.0 and 3.1, on Ruby 3.2 I'm getting this:

warning: LoadError: cannot load such file -- natalie_parser/natalie_parser

Accidentally found this issue:

echo 'p 1' > test-crlf.rb
todos test-crlf.rb 
bin/natalie test-crlf.rb 

This results in an error:

lib/natalie/parser.rb:32:in `parse': 1: syntax error, unexpected '\r' (SyntaxError)

An octal with a leading zero such as 0777 is represented as decimal 777, as opposed to an octal with a leading 0o, e.g. 0o777 which does get parsed properly as decimal value 511

Not 100% sure if this is a natalie_parser issue or Natalie issue

While working on natalie-lang/natalie#565 I noticed, that NatalieParser throws a SyntaxError with endless ranges as a last statement in a case/when block.

Example:

case 1
when 1
  1..
when 2
  2..
else
  3..
end

syntax error, unexpected 'when' (expected: 'end-of-line')

I tried to investigate but I'm not sure how to fix it properly. What I noticed is that when parsing the range, we consume the newline that separates the range expression from the the next when (or else) keyword. This leads to next_expression (being called by parse_case_when_body) to throw because when (or else) is not defined as being a possible end of an expression. One easy fix is adding those two keywords to the list defined in Token::is_end_of_expression() but I'm not feeling like this is a proper solution? The other idea would be to avoid consuming the newline that separates the range expression and the following keyword.

What solution should we aim to implement?

Hello there,
I recently noticed this new parser project.

It is quite an impressive feat to support most of the Ruby 3.0 syntax with a manually-written parser.

We're looking for a new parser for TruffleRuby, because maintaining a copy of the JRuby parser is proving to be quite a lot of effort. We are also considering sharing the parser, unmodified, with JRuby but that seems difficult as the JRuby parser currently relies on quite a few JRuby internals, jruby/jruby#6880.

To use this parser for TruffleRuby we would need it to be feature-complete, and for instance all the TODOs in https://github.com/natalie-lang/natalie_parser#to-do.
To be clear I'm not asking any of that, I'm just clarifying the requirements if TruffleRuby would use it.

I filed this issue to let you know about the potential interest.

Also I heard that @kddnewton might also consider making his own manually-written parser in C for Ruby, maybe @kddnewton can elaborate a bit on that?

cc @chrisseaton

Code:

def foo(id = ($A += 1) | 2)
  puts id
end
$A = 3
foo()
foo(777)

08:23 $ ruby -I lib:ext -r natalie_parser -e 'p NatalieParser.parse(File.read("../temp_natalie_parser_tests/args_oper.rb"))'
-e:1:in `parse': (string)#1: syntax error, unexpected '|' (expected: 'args closing paren') (SyntaxError)
def foo(id = ($A += 1) | 2)
                       ^ here, expected 'args closing paren'
        from -e:1:in `<main>'

I'm working to get ChunkyPNG to compile with Natalie, but first I need to get it to parse. :-)

[1,2,3].each do
  |x| puts x
end

The |x| on the next line is confusing the parser.

08:21 $ ruby -I lib:ext -r natalie_parser -e 'p NatalieParser.parse(File.read("../temp_natalie_parser_tests/each_nextline.rb"))'
-e:1:in `parse': (string)#2: syntax error, unexpected '|' (expected: 'expression') (SyntaxError)
  |x| puts x
  ^ here, expected 'expression'
        from -e:1:in `<main>'