Error in sa_*_translator.cpp(536): error C2039 when using ANTLR labled rules about speedy-antlr-tool

[amykyta3]

I tried on a smaller testcase but I couldn't reproduce the issue.
Can you confirm the following for me:

• What version of Antlr are you using to generate the parsers? I'm using 4.9.3
• What version of speedy-antlr-tool are you using? Latest is 1.2.0
• What version of Python are you using?
• Are you able to share the full g4 grammar file?

[amykyta3]

I think I see what is happening. I was able to re-generate the output with the grammar files you sent and I do not see the issue.

When you invoke Antlr to generate the two versions of the parser, both Python and C++ outputs need to be generated from the exact same g4 files.

It looks like your C++ parser was generated from JavaParserLabeled.g4 you sent me (after you added labels but before you renamed it to add the "Labeled" suffix in the grammar's name). The C++ parser output contains context classes that match the g4 files you sent me (has Statement1Context, Statement2Context, etc..)

I suspect the Python parser was generated from the original JavaParser.g4, before you modified it to add labels. Looking at the output Antlr generated, it is missing the labeled classes (only contains StatementContext).

The speedy-antlr-tool uses the Python output as a reference in order to generate the translator C++. If the Python and C++ targets were not generated from identical grammars, then the translator will have mismatches.

Try re-generating the Python parser target from Antlr using your latest grammar file. I suspect it will work.

For reference, here is the script I used:

#!/bin/bash
cd "$( dirname "${BASH_SOURCE[0]}" )"

antlr4="java -Xmx500M -cp /usr/local/lib/antlr-4.9.2-complete.jar org.antlr.v4.Tool"

# Generate C++ target with visitor
$antlr4 -Dlanguage=Cpp -o cpp_src JavaLexer.g4
$antlr4 -Dlanguage=Cpp -visitor -listener -o cpp_src JavaLabeledParser.g4

# Generate Python target
$antlr4 -Dlanguage=Python3 -o . JavaLexer.g4
$antlr4 -Dlanguage=Python3 -visitor -listener -o . JavaLabeledParser.g4

# Run speedy-antlr-tool to generate parse accelerator
python3 <<EOF
from speedy_antlr_tool import generate

generate(
    py_parser_path="JavaLabeledParser.py",
    cpp_output_dir="cpp_src",
)
EOF

[amykyta3]

My mistake. I copied the wrong text. Yes I renamed the grammar to JavaLabeledParser for it to work.
I have corrected my script above

[amykyta3]

When I originally developed the tool, it only supported grammars where the parser & lexer were in the same file. In that situation, the grammar would be simply named Java.g4, and Antlr would automatically append *Lexer and *Parser to the generated output. It was easier to keep this assumption when adding support for split grammars (#5) which is why the restriction exists. Not a great reason, but that's why it happened. If I'm bored I'll refactor and remove that assumption 😄

[m-zakeri]

The ANTLR rules, .g4 file, for the above code are as follows:

statement
    : blockLabel=block #statement0
    | ASSERT expression (':' expression)? ';' #statement1
    | IF parExpression statement (ELSE statement)? #statement2
    | FOR '(' forControl ')' statement #statement3
    | WHILE parExpression statement #statement4
    | DO statement WHILE parExpression ';' #statement5
    | TRY block (catchClause+ finallyBlock? | finallyBlock) #statement6
    | TRY resourceSpecification block catchClause* finallyBlock? #statement7
    | SWITCH parExpression '{' switchBlockStatementGroup* switchLabel* '}' #statement8
    | SYNCHRONIZED parExpression block #statement9
    | RETURN expression? ';' #statement10
    | THROW expression ';' #statement11
    | BREAK IDENTIFIER? ';' #statement12
    | CONTINUE IDENTIFIER? ';' #statement13
    | SEMI #statement14
    | statementExpression=expression ';' #statement15
    | identifierLabel=IDENTIFIER ':' statement #statement16
    ;

[m-zakeri]

Does anybody have any recommendations regarding this issue?

[amykyta3]

I was able to reproduce it on an older version of speedy-antlr-tool (v1.0.0)
Please upgrade to v1.2.0 and it should be fixed.

[m-zakeri]

I am also using speedy-antlr-tool (v1.2.0) and ANTLR 4.9.3.
I use Python 3.8 on Windows 10 (x64) with the MVC++ compiler from Visual Studio 2019.

I think the problem is due to improper handling of labeled statements in ANTLR grammars (e.g., # xxx) by speedy-antlr-tool (v1.2.0).

I attached my grammars files.

grammars.zip

[m-zakeri]

All generated files by speedy-antlr-tool (v1.2.0) for the above grammars:
parser.zip

[m-zakeri]

Thanks for your response. Actually, I get the following error with your script:

Traceback (most recent call last):
  File "speedya.py", line 3, in <module>
    generate(
  File "D:\Anaconda3\envs\Py38\lib\site-packages\speedy_antlr_tool\main.py", line 81, in generate
    raise ValueError("File does not look like a parser: %s" % py_parser_path)
ValueError: File does not look like a parser: JavaParserLabeled.py

How your script is run without error (I see that you did not rename the grammar)?

[m-zakeri]

Thanks for your efforts and time. It now works fine for me.
I also had to change the name of JavaLexer.g4 to JavaLabledLexer.g4.
Is there any reason for such a strict naming convention in grammar's names (e.g., ending the file name with *parser)?