Question: What's the recommended way of dealing with comments? about gogll

[goccmack]

Hi Ed,

I have so far only used gogll for domain specific languages in which I write the program in a markdown file in the spirit of Don Knuth's web and weave. I have intended to add ignored tokens to the lexer specification (in the style of github.com/goccmack/gocc). That would allow you to specify tokens to be suppressed by the lexer.

Your suggestion of providing a custom function in lexer.New() is elegant and very a simple change to gogll. But perhaps it is inconvenient to deal with lexical issues outside the grammar specification.

I find specifying every place in the grammar where comments are allowed obfuscates the grammar, unless you need the comments for code generation.

What are your thoughts on the alternatives? If you need this urgently I can quickly add the custom function to lexer.New() and return to suppressed tokens later.

[goccmack]

@EdSchouten
gogll v3.2.0 supports token suppression by the generated lexer. See gogll.md for details.

This can be used to implement code comments.

[goccmack]

The updated example is on github. You will see that I fixed the patterns for both the block and line comments. And the test.txt file with such a c-style comment is now also included.

…

On 12 Aug 2020, at 17:30, Ed Schouten ***@***.***> wrote: Nice! I've just given it a try and it seems to work. One limitation that I've noticed is that the provided example: !block_comment : '/''*' {not "*"} '*''/' ; doesn't allow parsing these kinds of comments that you often see in C-like programming languages: /* * This is a block comment. */ I guess that a non-greedy matching operator (a la regex .*?) is needed to support constructs like those. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZRQM4ATOWXZFBZA4GLCSTSAKYS7ANCNFSM4PZHDC2A>.

[EdSchouten]

I was merely using gogll as an experiment to keep myself busy on a Sunday. There’s no need to implement anything quickly just for me. I’m happy with the solution that you think is best!

[goccmack]

OK. Having thought about it I still prefer suppressed tokens because it keeps the complete lexical specification in one place. I will implement it as soon as I have the time.

[EdSchouten]

Nice! I've just given it a try and it seems to work. One limitation that I've noticed is that the provided example:

!block_comment : '/''*' {not "*"} '*''/' ;

doesn't allow parsing these kinds of comments that you often see in C-like programming languages:

/*
 * This is a block comment.
 */

I guess that a non-greedy matching operator (a la regex .*?) is needed to support constructs like those.

[goccmack]

You are right. I was focussed on the code generation and not the example. When I went out for boodschappen I immediately realised the limitation of the pattern. I will update it and post it.

…

On 12 Aug 2020, at 17:30, Ed Schouten ***@***.***> wrote: Nice! I've just given it a try and it seems to work. One limitation that I've noticed is that the provided example: !block_comment : '/''*' {not "*"} '*''/' ; doesn't allow parsing these kinds of comments that you often see in C-like programming languages: /* * This is a block comment. */ I guess that a non-greedy matching operator (a la regex .*?) is needed to support constructs like those. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZRQM4ATOWXZFBZA4GLCSTSAKYS7ANCNFSM4PZHDC2A>.