Simple tokenizer question about superpower

[nblumhardt]

Hi!

The tokenizer will consider the whole span matched by the parser to be the token, regardless of the value that's returned, but, you can use .Apply(HashIdent) later on in the parsing stage to get the value, i.e. Token.EqualTo(Tokens.Identifier).Apply(HashIdent).

For this to work you need to drop off the Value(Unit.Value) piece:

static TextParser<Unit> HashIdent { get; } = Character.EqualTo('#').IgnoreThen(Identifier.CStyle)

HTH!

[dhowe]

Thanks, thats very helpful (I was just wondering what the apply was doing in your example code)!

When I do that, I get a compile error... do I need to change the generic type to TextSpan?

[dhowe]

Also, how do I handle the opposite case, where I want to ignore the last character?

        static TextParser<Unit> Actor { get; } =
            from name in Character.LetterOrDigit.Many()
            from last in Character.EqualTo(':')
            select Unit.Value;

[AndrewSav]

Yes, dropping Unit means that you are returning a different value now. If value is not needed for performance reasons it's better to use unit, but in your case change it to TextSpan as you suggested.

As for your second question something like this would probably work:

static TextParser<string> Actor { get; } =
			from name in Character.LetterOrDigit.Many()
			from last in Character.EqualTo(':')
			select new string(name);

[nblumhardt]

I guess this water is long under the bridge - hope you found a good solution, @dhowe - closing as stale, but keep us posted :-)