`bad-string-token` with CRLF needs to advance two code points.

#9 · open · 0 comments

View on GitHub ↗

romainmenke

https://github.com/keithamus/csslex/blob/7182786dfcd1a553ac60f82d7a66bb4a48607d29/mod.ts#L338-L339 Example: `"foo\r\nbar"` The spec actually [requires replacing `\r\n` with `\n`](https://drafts.csswg.org/css-syntax/#input-preprocessing) but that isn't practical for dev tools that should support lossless tokenization and serialization. A result of not doing this preprocessing is that some newlines are actually two codepoints long, instead of just one. When a `bad-string-token` is produced as the result of encountering `\r`, the tokenizer needs to check if the next position is `\n` and then advance one more position.

Comments