jruby / jruby-parser

JRuby's parser customized for IDE usage

JRubyParser.jar and jruby-parser.gem

JRuby once had a parser which kept track of all sorts of extra information when it built it's Abstract Syntax Tree (AST). Stuff like character offsets where a particular element started or ended. The impact of this extra information was a more than noticeable amount of memory and a bit of a perf impact. At the time we decided to discontinue having this sort of parser in JRuby we created JRubyParser.

JRubyParser.java is just the Java code which is slowly evolving into everything a Ruby IDE project could want. Ability to know where source elements are; whether a syntax is correct; source re-writing....

Netbeans and Eclipse are two users of JRubyParser. We have a vested interest in making parsing Ruby a convenient and simple task for Java programmers.

Basic Usage

Java

import org.jrubyparser.CompatVersion;
import org.jrubyparser.Parser;
import org.jrubyparser.ast.*;
import org.jrubyparser.parser.ParserConfiguration;
import java.io.StringReader;

public class ParseSomething {

    public static void main(String[] args) {
        String codeString = "def foo(bar)\n bar\n end\n foo('astring')";

        Node node = parseContents(codeString);
        System.out.println(node);
    }

    public static Node parseContents(String string) {
        Parser rubyParser = new Parser();
        StringReader in = new StringReader(string);
        CompatVersion version = CompatVersion.RUBY1_8;
        ParserConfiguration config = new ParserConfiguration(0, version);
        return rubyParser.parse("<code>", in, config);
    }
}

In the above code, CompatVersion is an enum which can be RUBY1_8,RUBY1_9, RUBY2_0, or RUBY2_3. ParserConfiguration takes several options, including linenumber, version, and optionally a static scope and can be configured with additional information about syntax gathering if necessary.

Ruby

jruby-parser.gem is a gem which bundles JRubyParser.jar and also provides a thin Ruby layer around the Java API to follow Ruby programming idioms better. Here is a simple example of parsing and rewriting using jruby-parser.rb:

require 'jruby-parser'

root = JRubyParser.parse("b = foo(1)")
fcall = root.find_node(:fcall)
fcall.name = 'bar'
fcall.args[0] = true

# Write out the new source 
root.to_source # b = bar(true)

Building

Generate new parser code (only needed on hacking .y files):

Assume: jay 1.0.2 installed

./bin/generate_parser Ruby18Parser Ruby18
./bin/generate_parser Ruby19Parser Ruby19
./bin/generate_parser Ruby20Parser Ruby20

Build and test using Rake

Compiles .java files, builds jar, copies jar to lib directory and runs the specs:

jruby -S rake

Builds gemfile:

jruby -S rake build

Runs specs:

jruby -S rake spec

Build and package using Maven to get a JAR

mvn compile
mvn package

jruby-parser's Issues

The Maven POM version is incorrect.

The version in the Maven POM is still at 0.2 although this version has been released to the general public. There has been significant API changes since then. This leads to problems when using the source since Maven has no way of telling the latest source apart from the released artifact.

Please follow standard patterns and bump the version (for instance to 0.3-SNAPSHOT) as soon as a release has been made. The 0.2 version with no suffix is supposed to be immutable.

2.0 parser sometimes reports iters as args

callcc { |c| c.call }

1.9 parser, getArgs ⇒ nil, getIter ⇒ { |c| c.call }
2.0 parser, getArgs ⇒ { |c| c.call }, getIter ⇒ nil

Workaround is to look for an iter node in your args, which shouldn't be there without a block pass.

Another casualty of the nil elimination perhaps?

Misses empty expressions in literal array

Parses this:

[0, (), 2, (), 4]

as this:

(RootNode, (NewlineNode, (ArrayNode, (FixnumNode), (FixnumNode), (FixnumNode))))

which is wrong according to this:

https://github.com/rubyspec/rubyspec/blob/master/language/array_spec.rb#L13

Perhaps due to the work to remove the implicit nil nodes?

Feature Request: Create AbstractNodeVisitor

Can you please consider the creation of an "AbstractNodeVisitor" class that implements all methods in NodeVisitor interface to return null? This simplifies the process of writing a NodeVisitor as we only need to implement the methods that we care about.

Implement the 2.0 parser

I know it's a lot of work, but I'll open this issue to show my hope!

Next, Break, Rescue, RescueBody, Module have inconsistent asserts

For example, NextNode asserts that the value is not null (https://github.com/jruby/jruby-parser/blob/master/src/org/jrubyparser/ast/NextNode.java#L45), but the parser creates such nodes with value as null (https://github.com/jruby/jruby-parser/blob/master/src/org/jrubyparser/parser/Ruby19Parser.y#L1202).

There's the same problem with at least Break, Rescue, RescueBody and Module.

Is it the assert which is wrong, or the parser? I would guess that it is the assert which is wrong.

Do we need to run the specs with assertions enabled to see these bugs?

Parses != as !(x==y)

jruby-parser seems to parse x!=y as !(x==y). That is, this:

(q1 = p[1]) != 1

is parsed as:

(NotNode, (CallNode:==, (NewlineNode, (LocalAsgnNode:q1,
        (CallNode:[], (LocalVarNode:p), (ArrayNode,
        (FixnumNode))))), (ArrayNode, (FixnumNode))

`not` is parsed as an operator

https://github.com/jruby/jruby-parser/blob/master/src/org/jrubyparser/parser/Ruby19Parser.y#L439

c.f. kAND and kOR

Ruby 2.0 parser fails to parse lambda arrow syntax with space

It seems like there is an inconsistency introduced in ruby 2.0 regarding the use of lambda syntax. The issue is detailed here

In ruby 2.0 the following is valid (notice the space between -> and (message))

hello_world = -> (message) { puts message }

However in ruby 1.9 it gives an error.

The issue is that in jruby-parser this doesn't parse even with CompatVersion set to 2.0. And in the older versions it won't parse, so the files using this cannot be handled with the jruby-parser.

Removing the space between the arrow and the parentheses would allow jruby-parser to parse these files. But there are uses of this syntax (with spaces) in the wild already, e.g. in slop gem. So, there are many files using this syntax that cannot be handled with jruby-parser.

IndexOutOfBounds with an array node

Hi, I am parsing this code:

'a,@b,c = 1,2,3'

with:

r = JRubyParser.parse(code)
man = r.childNodes[0].childNodes[0]

Now, if I invoke man.pre.count I get a 4 but if I later try to get man.pre.get(3) it blows up. Any ideas about that? Am I using it uncorrectly?

Fails in lexer with a file using the unicode characters

The parser fails to parse a file containing unicode characters like the following :

  class Queue

    def clear
    end
    alias_method :💣, :clear
  end

I made sure I am not having any character encoding issues by ensuring that the file is read properly using "UTF-8" encoding. We can see the character clearly when I print the file before calling the jruby-parser which throws an exception as shown below:

  class Queue

    def clear
    end
    alias_method :💣, :clear
  end

org.jrubyparser.lexer.SyntaxException
    at org.jrubyparser.lexer.Lexer.identifier(Lexer.java:1888)
    at org.jrubyparser.lexer.Lexer.yylex(Lexer.java:1478)
    at org.jrubyparser.lexer.Lexer.nextToken(Lexer.java:483)
    at org.jrubyparser.parser.Ruby20Parser.yyparse(Ruby20Parser.java:1515)
    at org.jrubyparser.parser.Ruby20Parser.yyparse(Ruby20Parser.java:1466)
    at org.jrubyparser.parser.Ruby20Parser.parse(Ruby20Parser.java:4666)
    at org.jrubyparser.Parser.parse(Parser.java:86)

This is not a test case I made up, I was actually trying to parse some real ruby source code e.g. it is used here. After I ran into this issue, I also tried a few other characters they also fail to parse:

  class Queue

    def clear
    end
    alias_method :☂, :clear
  end

org.jrubyparser.lexer.SyntaxException
    at org.jrubyparser.lexer.Lexer.identifier(Lexer.java:1888)
    at org.jrubyparser.lexer.Lexer.yylex(Lexer.java:1478)
    at org.jrubyparser.lexer.Lexer.nextToken(Lexer.java:483)
    at org.jrubyparser.parser.Ruby20Parser.yyparse(Ruby20Parser.java:1515)
    at org.jrubyparser.parser.Ruby20Parser.yyparse(Ruby20Parser.java:1466)
    at org.jrubyparser.parser.Ruby20Parser.parse(Ruby20Parser.java:4666)
    at org.jrubyparser.Parser.parse(Parser.java:86)

Parser error with required keyword arguments

The following code gives no syntax error when using ruby, but fails to parse with jruby parser (in all compat modes).

     def initialize(access_key_id:, secret_access_key:, bucket:, max_size: nil, prefix: nil, hasher: Refile::RandomHasher.new)
        @access_key_id = access_key_id
        @secret_access_key = secret_access_key
        @s3 = AWS::S3.new(access_key_id: access_key_id, secret_access_key: secret_access_key)
        @bucket_name = bucket
        @bucket = @s3.buckets[@bucket_name]
        @hasher = hasher
        @prefix = prefix
        @max_size = max_size
      end

Parsing rescue statements is wrong for lexical representation

  begin
    puts "test"
  rescue Exception => hello
    puts hello
  end

This snippet will create a block node and then for hello it will make a localasgnnode and the value will be $!. We definitely do not want the AST to contain $! since it is not actually in the source and we cannot neccesarily use localasgnnode since it implies assignment. We do need to declare this as a var for things to properly parse, so I think this is implying a new type of node perhaps.

Some variants of the || operator are parsed like calls

It looks like some variants of the || operator come out as if they are calls, when they should only ever be special 'or' nodes.

For example this:

foo[bar] ||= baz

Parses as this:

(RootNode,
    (NewlineNode,
        (OpElementAsgnNode,
            (VCallNode:foo),
            (ArrayNode, (VCallNode:bar)),
        (VCallNode:baz))))

With the operator name of OpElementAsgnNode being "||". That makes it look like a call, when it's something else entirely. Unless you look specifically for "||", you're going to assume that this is a call and not realise until the "||" method isn't found later.

I'd suggest that we have a special version of OpElementAsgnNode for And and Or, something like OpElementAsgnOrNode, like we do with OpAsgnOrNode.

ReWriteVisitor Generates Incorrect Syntax For "alias" and "def"

An attempt to parse the AST and write the AST back to disk using ReWriteVisitor produces incorrect syntax for aliases and function declarations. The right-hand side of the alias expression seems to be missing and arguments for function declarations seem to be skipped. Also, a period is showing up after some identifiers.

== sample before ==

class ActionController::Base
  alias orig_params params
  def params
    if orig_params != nil && !orig_params.respond_to?("in")
      orig_params.extend(ActionController_Parameters_Mixin)
    end

    return orig_params
  end

  alias orig_redirect_to redirect_to
  def redirect_to(options={}, response_status={})
    if options.kind_of? String

    elsif options.kind_of? Hash

    else
      # give up
    end

    orig_redirect_to options, response_status
  end
end

== rewritevisitor result ==

class ActionController::Base
  alias  
  def params
    if orig_params.!=(nil) and orig_params.respond_to?("in").!
      orig_params.extend(ActionController_Parameters_Mixin)
    end
    return orig_params
  end
  alias  
  def redirect_to , 
    if options.kind_of? String
    elsif options.kind_of? Hash
    end
    orig_redirect_to(options, response_status)
  end
end

Note the "alias" statements, the arguments for defs (ex: "redirect_to") and the extra period after "orig_params" in the "def params" function.

Push a gem release?

Really trying to finally get rsense out the door and noticed the current gem is a bit behind. I can work from a local copy of course, but that's not going to work for folks installing it. Would appreciate this.
Thanks.

Rewrite code case

I am using jruby-parser to rewrite code, but with some code is has some problems:
This:
require 'jruby-parser'

root = JRubyParser.parse( %q{
def log(severity)
case severity
when Severity::DEBUG
@logging_services.debug(msg.to_s)
when Severity::INFO
@logging_services.info(msg.to_s)
when Severity::WARNING
@logging_services.warn(msg.to_s)
else
@logging_services.error(msg.to_s)
end
end
} )

root.to_source

produces:
def log severity
case severity[
when Severity
@logging_services.debug(msg.to_s)
when Severity
@logging_services.info(msg.to_s)
when Severity
@logging_services.warn(msg.to_s)
else
@logging_services.error(msg.to_s),
when Severity
@logging_services.info(msg.to_s)
when Severity
@logging_services.warn(msg.to_s)
else
@logging_services.error(msg.to_s),
when Severity
@logging_services.warn(msg.to_s)
else
@logging_services.error(msg.to_s)]
end
end

Can I do something to avoid this behaviour?

StringTerm.parseString doesn't properly handle nested string interpolation with string containing '}'

This is a hard one to describe without code, so here goes:

string = "here's some code: #{var = '}'; 1} there"

In this case, if we set up StringTerm to split embedded tokens, it will pick up the first } as the end of the dynamic expression, rather than the last one.

My best guess is that we'd read until the end of the string, then unread to the last unescaped '}' and use that as the end - but of course we'd have to worry about multiple dynamic expressions in the same string breaking that.

Perhaps a simple stack of quotes to help avoid it: match up pairs of single quotes, double quotes, /, so that if we encounter } inside we ignore?

Beginners question

Hi,
I've stumbled upon JRubyParse because I need to do some static analysis on huge ruby on rails projects(~1000 ruby files). I need to extract all defined classes+ defined methods and I need to make a function call graph (not visually, as a list). So basically I need to recognize classes/modules, functions, and also function calls. I wanted to ask if you think it is possible/makes sense doing this with your parser?
Also is there any documentation on how to use it? How can I rewrite? I'm planning to use the java jar.

Edit: To also make a concrete question, when the parser visits/recognizes a FCallNode how can I extract function name and parameters(to afterwards save them as string)?

Thanks for reading and kind regards,

Alexander

Need a way to pass in an existing local scope

We need a way to parse Ruby code in an existing local scope, so with existing local variables.

For example, parsing the expression a will generate a method call by default. I'd like to be able to tell jruby-parser that a is an existing local variable, and get it to generate a local variable read.

This has at least two applications - parsing code to be eval'd in a Binding, and parsing lines of a REPL.

A workaround is to add dummy assignments to the local variables in your scope, and then remove them later.

Implement stubby lambda node (1.9)

https://github.com/jruby/jruby-parser/blob/master/src/org/jrubyparser/ast/LambdaNode.java#L58

Doesn't parse hash literal with = at end of key name

Under MRI 1.8, 1.9, 2.0, JRuby:

irb(main):001:0> {:a==>1}
=> {:a==>1}

jruby-parser:

> {:a==>1}
syntax error, unexpected tGT

Spec:

https://github.com/rubyspec/rubyspec/blob/master/language/hash_spec.rb#L55

IndexOutOfBoundsException while rewriting

I parsed this method and I was trying to regenerate the code with this code:

            sw = java.io.StringWriter.new
            rwv = org.jrubyparser.rewriter.ReWriteVisitor.new(sw,'')
            cbw = org.jrubyparser.rewriter.ClassBodyWriter.new(rwv,corr_node)
            cbw.write
            sw.to_string

This code works for thousand of methods but fail on a few.

The methods parsed and failed 1/2:

     def index
          @question = params[:q] || ""
          @question.strip!
          @all_words = params[:all_words] ? params[:all_words].present? : true
          @titles_only = params[:titles_only] ? params[:titles_only].present? : false

          projects_to_search =
            case params[:scope]
            when 'all'
              nil
            when 'my_projects'
              User.current.memberships.collect(&:project)
            when 'subprojects'
              @project ? (@project.self_and_descendants.active.all) : nil
            else
              @project
            end

          offset = nil
          begin; offset = params[:offset].to_time if params[:offset]; rescue; end

          # quick jump to an issue
          if (m = @question.match(/^#?(\d+)$/)) && (issue = Issue.visible.find_by_id(m[1].to_i))
            redirect_to issue_path(issue)
            return
          end

          @object_types = Redmine::Search.available_search_types.dup
          if projects_to_search.is_a? Project
            # don't search projects
            @object_types.delete('projects')
            # only show what the user is allowed to view
            @object_types = @object_types.select {|o| User.current.allowed_to?("view_#{o}".to_sym, projects_to_search)}
          end

          @scope = @object_types.select {|t| params[t]}
          @scope = @object_types if @scope.empty?

          # extract tokens from the question
          # eg. hello "bye bye" => ["hello", "bye bye"]
          @tokens = @question.scan(%r{((\s|^)"[\s\w]+"(\s|$)|\S+)}).collect {|m| m.first.gsub(%r{(^\s*"\s*|\s*"\s*$)}, '')}
          # tokens must be at least 2 characters long
          @tokens = @tokens.uniq.select {|w| w.length > 1 }

          if [email protected]?
            # no more than 5 tokens to search for
            @tokens.slice! 5..-1 if @tokens.size > 5

            @results = []
            @results_by_type = Hash.new {|h,k| h[k] = 0}

            limit = 10
            @scope.each do |s|
              r, c = s.singularize.camelcase.constantize.search(@tokens, projects_to_search,
                :all_words => @all_words,
                :titles_only => @titles_only,
                :limit => (limit+1),
                :offset => offset,
                :before => params[:previous].nil?)
              @results += r
              @results_by_type[s] += c
            end
            @results = @results.sort {|a,b| b.event_datetime <=> a.event_datetime}
            if params[:previous].nil?
              @pagination_previous_date = @results[0].event_datetime if offset && @results[0]
              if @results.size > limit
                @pagination_next_date = @results[limit-1].event_datetime
                @results = @results[0, limit]
              end
            else
              @pagination_next_date = @results[-1].event_datetime if offset && @results[-1]
              if @results.size > limit
                @pagination_previous_date = @results[-(limit)].event_datetime
                @results = @results[-(limit), limit]
              end
            end
          else
            @question = ""
          end
          render :layout => false if request.xhr?
        end

methods 2/2:

  def pretty_title
        WikiPage.pretty_title(title)
  end

This is the exception I get:

String.java:695:in charAt': java.lang.StringIndexOutOfBoundsException: String index out of range: 1344 from ShortIfNodeReWriteVisitor.java:47:invisitNewlineNode'
from NewlineNode.java:65:in accept' from ReWriteVisitor.java:247:invisitNode'
from ReWriteVisitor.java:942:in printShortIfStatement' from ReWriteVisitor.java:968:invisitIfNode'
from IfNode.java:61:in accept' from ReWriteVisitor.java:247:invisitNode'
from ReWriteVisitor.java:1146:in visitNewlineNode' from NewlineNode.java:65:inaccept'
from ReWriteVisitor.java:247:in visitNode' from ReWriteVisitor.java:1569:invisitNodeInIndentation'
from ReWriteVisitor.java:1579:in visitWhenNode' from WhenNode.java:59:inaccept'
from ReWriteVisitor.java:247:in visitNode' from ReWriteVisitor.java:1581:invisitWhenNode'
from WhenNode.java:59:in accept' from ReWriteVisitor.java:247:invisitNode'
from ReWriteVisitor.java:1581:in visitWhenNode' from WhenNode.java:59:inaccept'
from ReWriteVisitor.java:247:in visitNode' from ReWriteVisitor.java:334:invisitAndPrintWithSeparator'
from ReWriteVisitor.java:420:in visitArrayNode' from ArrayNode.java:59:inaccept'
from ReWriteVisitor.java:247:in visitNode' from ReWriteVisitor.java:596:invisitCaseNode'
from CaseNode.java:74:in accept' from ReWriteVisitor.java:247:invisitNode'
from ReWriteVisitor.java:1602:in visitNewlineInParentheses' from ReWriteVisitor.java:900:inprintAsgnNode'
from ReWriteVisitor.java:1075:in visitLocalAsgnNode' from LocalAsgnNode.java:62:inaccept'
from ReWriteVisitor.java:247:in visitNode' from ReWriteVisitor.java:1146:invisitNewlineNode'
from NewlineNode.java:65:in accept' from ReWriteVisitor.java:247:invisitNode'
from ReWriteVisitor.java:255:in visitIter' from ReWriteVisitor.java:457:invisitBlockNode'
from BlockNode.java:54:in accept' from ReWriteVisitor.java:247:invisitNode'
from ReWriteVisitor.java:758:in printDefNode' from ReWriteVisitor.java:767:invisitDefnNode'
from DefnNode.java:52:in accept' from ReWriteVisitor.java:247:invisitNode'
from ReWriteVisitor.java:1146:in visitNewlineNode' from NewlineNode.java:65:inaccept'
from ReWriteVisitor.java:247:in visitNode' from ReWriteVisitor.java:1569:invisitNodeInIndentation'
from ClassBodyWriter.java:28:in write' from null:-1:ininvoke'
from DelegatingMethodAccessorImpl.java:43:in invoke' from Method.java:601:ininvoke'
from JavaMethod.java:440:in invokeDirectWithExceptionHandling' from JavaMethod.java:304:ininvokeDirect'
from InstanceMethodInvoker.java:52:in call' from CachingCallSite.java:134:incall'
from models_redundancy_calc.rb:43:in block_5$RUBY$model_handler' from models_redundancy_calc$block_5$RUBY$model_handler:-1:incall'
from CompiledBlock19.java:163:in yield' from CompiledBlock19.java:91:incall'
from Block.java:89:in call' from RubyProc.java:261:incall'
from RubyProc.java:249:in call19' from RubyProc$INVOKER$i$0$0$call19.gen:-1:incall'
from DynamicMethod.java:209:in call' from DynamicMethod.java:205:incall'
from CachingCallSite.java:168:in call' from models_redundancy_calc.rb:55:inblock_6$RUBY$model_handler'
from models_redundancy_calc$block_6$RUBY$model_handler:-1:in call' from CompiledBlock19.java:139:inyield'
from Block.java:130:in yield' from RubyArray.java:1606:ineachCommon'
from RubyArray.java:1613:in each' from RubyArray$INVOKER$i$0$0$each.gen:-1:incall'
from CachingCallSite.java:143:in callBlock' from CachingCallSite.java:154:incallIter'
from models_redundancy_calc.rb:49:in block_1$RUBY$model_handler' from models_redundancy_calc$block_1$RUBY$model_handler:-1:incall'
from CompiledBlock19.java:163:in yield' from CompiledBlock19.java:91:incall'
from Block.java:89:in call' from RubyProc.java:261:incall'
from RubyProc.java:249:in call19' from RubyProc$INVOKER$i$0$0$call19.gen:-1:incall'
from DynamicMethod.java:217:in call' from DynamicMethod.java:213:incall'
from CachingCallSite.java:202:in call' from CallTwoArgNode.java:59:ininterpret'
from NewlineNode.java:105:in interpret' from BlockNode.java:71:ininterpret'
from IfNode.java:118:in interpret' from NewlineNode.java:105:ininterpret'
from BlockNode.java:71:in interpret' from ASTInterpreter.java:75:inINTERPRET_METHOD'
from InterpretedMethod.java:290:in call' from DefaultMethod.java:228:incall'
from CachingCallSite.java:245:in `callBlock'

mvn package fails with compilation error.

Cloned the repo and ran mvn clean package. The build process failed with the following compilation error.

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /Users/sgopal1/code/jruby-parser/src/org/jrubyparser/rewriter/ReWriteVisitor.java:[168,7] error: ReWriteVisitor is not abstract and does not override abstract method visitRequiredKeywordArgumentValueNode(RequiredKeywordArgumentValueNode) in NodeVisitor
[ERROR] /Users/sgopal1/code/jruby-parser/src/org/jrubyparser/util/NoopVisitor.java:[124,7] error: NoopVisitor is not abstract and does not override abstract method visitRequiredKeywordArgumentValueNode(RequiredKeywordArgumentValueNode) in NodeVisitor
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.634 s
[INFO] Finished at: 2017-04-04T13:37:09-08:00
[INFO] Final Memory: 18M/389M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project jrubyparser: Compilation failure: Compilation failure:
[ERROR] /Users/sgopal1/code/jruby-parser/src/org/jrubyparser/rewriter/ReWriteVisitor.java:[168,7] error: ReWriteVisitor is not abstract and does not override abstract method visitRequiredKeywordArgumentValueNode(RequiredKeywordArgumentValueNode) in NodeVisitor
[ERROR] /Users/sgopal1/code/jruby-parser/src/org/jrubyparser/util/NoopVisitor.java:[124,7] error: NoopVisitor is not abstract and does not override abstract method visitRequiredKeywordArgumentValueNode(RequiredKeywordArgumentValueNode) in NodeVisitor
[ERROR] -> [Help 1]

Parser duplicating nodes

Iterating through the nodes of a jruby-parser AST will occasionally yield identical nodes twice. I'm trying to parse this file and identify function calls:

require 'jruby-parser'

root = JRubyParser.parse(contents_of_relations_controller)
function_calls = root.select { |node| node.nodeType.to_s == "FCALLNODE" }
function_calls.size # => 35
function_calls.uniq(&:object_id).size # => 33

Why would the array of function calls contain the same object twice?

Why does the 1.8 parser pass a different name for unary operators than 1.9?

Compare line 821 of Ruby19Parser.y of 797 of Ruby18Parser.y. The 1.9 parser sets the correct name of the method, which is -@, whereas 1.8 doesn't set the name, so I think it takes the name from the token, "-".

This means that when I get an AST using the 1.9 parser, the methods which are actually -@ are reported as -, and there's no way to tell that they were ever unary (maybe guessing based on node locations or something).

Is this right? I don't think there's anything that changed between 1.8 and 1.9 that means that you'd want to treat unary methods differently?

2.0 parser doesn't give call with block nodes the entire source position

Referring to https://github.com/jruby/jruby-parser/blob/master/spec/positions/call_spec.rb#L100.

1.8 and 1.9 report the source position of these nodes as including the block, while 2.0 seems to report them as excluding the block. To see what I mean, change https://github.com/jruby/jruby-parser/blob/master/spec/positions/call_spec.rb#L4 to include 2.0 and you will see the relevant failures.

I can't immediately see what is causing this - they seem to use the same union method on the same nodes to get it.

ReWriteVisitor malforms code for argument nodes and symbols

When I pass a node two ReWriteVisior.createCodeFromNode, block arguments do not have separators and symbols do not have colons.

Sample Java code:

Parser rubyParser = new Parser();
String source = "method do |one, two|\n  {\n    :foo => bar,\n    :baz => bot\n  }\nend\n";
StringReader in = new StringReader(source);
CompatVersion version = CompatVersion.RUBY1_9;
ParserConfiguration config = new ParserConfiguration(0, version);
Node node = rubyParser.parse("<code>", in, config);
String dummy = "";
String result = ReWriteVisitor.createCodeFromNode(node, dummy);
System.out.println(result);

Expected output:

method() do |one, two|
  {:foo => bar, :baz => bot}
end

Actual output:

method() do |onetwo|
  {foo => bar, baz => bot}
end

I submitted a pull request that fixes both issues.

Invoke the 1.8 or 1.9 parser

Excuse my simple question but how do I specify which version of the parser do I want to use? Wouldn't be useful to have it in the README?