Multiple possible token types about participle

[alecthomas]

The lack of context does make some things a bit more difficult, that’s for sure. In the specific example of the SQL parser I’d probably just not make SELECT a keyword.

But generally, I’m not sure. I have solved this in one parser by pushing the statefulness down into the lexer. This was a parser for a nested string interpolation language eg. ”hello ${who “nested ${other}”}”. A similar approach can be used for indentation aware languages like Python. I don’t think this approach would be very applicable to the SQL example though as too much logic would have to be pushed down into the lexer.

I’m not against non-CFG but I don’t know if there’s an elegant way to express the grammar like EBNF?

[beebeeep]

just not make SELECT a keyword.

Well, it's still nice to have separate keyword type, in particular case - for case-insensitive keyword matching (btw, I've seen that pointlander/peg is using 'token' for case-sensitive matching and "token" for case-insensitive - a bit unobvious but quite elegant)

Regarding how this type ambiguity can be expressed in grammar, for me it seems like no changes here needed - if parser failed to parse current token with given type, it could ask lexer for other options. Or else lexer can have Token.Type as []rune, with standard parser return 1-element slice with option to create custom lexer with multiple choices.

[alecthomas]

Regarding how this type ambiguity can be expressed in grammar, for me it seems like no changes here needed - if parser failed to parse current token with given type, it could ask lexer for other options. Or else lexer can have Token.Type as []rune, with standard parser return 1-element slice with option to create custom lexer with multiple choices.

I don't think that's a general solution to contextual-grammars, it just solves one specific example. For example that would not work for the example of the nested interpolation grammar I gave.

[alecthomas]

btw, I've seen that pointlander/peg is using 'token' for case-sensitive matching and "token" for case-insensitive - a bit unobvious but quite elegant

I think this is a good idea; the current approach of case case normalisation isn't really a good one. I think I'll add a parser option to do this, eg. participle.CaseInsensitive("Keyword").

[beebeeep]

I've been thinking about this for a while - yeah, looks like subj has too narrow scope; also have no other ideas, unfortunately, so I'd rather close issue.

[alecthomas]

I've added the CaseInsensitive() option.