metaborg / spt Goto Github PK

The semantics of string escapes is confusing (to me) when specifying test expectations.

test stringescape [[ "" ]] parse to Mod(String("\"\""))

This test succeeds as expected.

test stringescape [[ "\"" ]] parse to Mod(String("\"\\\"\""))

This test fails unexpectedly. I checked the expected string by printing the input program in ATerm format, which prints the same string as in the expected test result.

The test passes when I delete one of the triplet of backslashes:

test stringescape [[ "\"" ]] parse to Mod(String("\"\\"\""))

The error message was not very helpful as it does not correctly print the expected result, not does the parse result it contains match the 'fixed' test result:

The fragment did not parse to the expected ATerm. Parse result was: Mod(String("\"\\\"\"")) Expected result was: Mod(String(\"\\\\\"\"))

Summary
According to the documentation, a single open square bracket ([) is allowed in test descriptions as it is not a marker. However, it does not work: the file fails to parse.

Todo

• Create syntax tests with a single open square bracket in the description
• Fix bug (see 'implementation details' below)

Reproduction steps
Given: An example test case from PIE, but it should happen with any language:

test name ends with [ [[ data Foo[ = foreign java org.example.test.Foo {} ]] parse fails

Expected: The SPT file parses
Actual: The SPT file does not parse, error at the first [: Syntax error, not expected here: '['

Implementation details
This is caused by line 137 of the grammar: Description = (~[\n\r\[])+
It does not allow any open square brackets in the description, while it should allow one.
This could be resolved by making a special case for a single [:

Description = (~[\n\r\[]|\[~[\[])+

Note: I am not completely sure that this will work as is

Currently you can write a test expectation run my-test-strat(|"some term args", 1) to [[ my language snippet ]] but not run my-test-strat(|"some term args", 1) to MyATerm(Expectation("hi?")) (or rather, the latter gets interpreted as a run expectation + a parse-to expectation).

Many languages allow programs to import programs from other files. To test things related to importing other files, you need to be able to specify multiple files in a single test case. However, currently you can specify at most one full program (file) in a single SPT test case. So, testing the import feature of languages is not possible with SPT.

A current workaround is adding a language construct to your language that replicates the start of a new file in a single file, but that workaround changes the grammar of a language purely for testing which is not tenable.