How to lex heredocs (Caddyfile) about chroma

[alecthomas]

If it's already in Go then it's definitely fine to update.

I find that surprising. Why is compiled Go code slower/larger?

This is a bit surprising, but each lexer is a whole bunch of what is effectively dynamic code to the Go compiler, so it a) expands to a very large amount of machine code and b) takes a considerable amount of time to execute at init time. If the Go compiler were a lot smarter this would probably end up being static data sections, but it isn't.

I can't find the heredoc lexing. Can you point me to it? I don't see any << stuff in bash.xml.

Search for <<. It's a single regex with a backreference.

[alecthomas]

Look at how the bash lexer does it, it's the same concept.

I will not accept Go lexers unless there's a good reason, you should use XML. Chroma is used by a lot of people who aren't programmers, so XML is a better choice to foster contributions than Go. It's also much much more efficient in terms of binary size and startup time.

[francislavoie]

I will not accept Go lexers unless there's a good reason, you should use XML.

To be clear, I'm not making a new lexer, only updating an existing one. https://github.com/alecthomas/chroma/blob/master/lexers/caddyfile.go

It's also much much more efficient in terms of binary size and startup time.

I find that surprising. Why is compiled Go code slower/larger?

Look at how the bash lexer does it, it's the same concept.

I can't find the heredoc lexing. Can you point me to it? I don't see any << stuff in bash.xml.

Edit: oh, <<< 🤦‍♂️

[francislavoie]

I see the \2. So it's a single regexp.

Is there a way to do this with state though (push + pop)? I have some other token types I want to handle within the heredoc.

For example, if something like {foo} appears within the heredoc, I want that to be a LiteralStringEscape.