Windows line endings cause IndexOutOfRange during tokenization about sqlglot

[georgesittas]

Hey @mr-miles, can you share a snippet that reproduces the issue here first?

[mr-miles]

Good idea!

from sqlglot import parse_one, exp

# find all tables (x, y, z)
for table in parse_one("SELECT * FROM x JOIN y JOIN z\r\n").find_all(exp.Table):
    print(table.name)

Blows up with qlglot.errors.TokenError

End the query with \r or \n on their own or some other whitespace and it is fine

[georgesittas]

I see, I can reproduce this issue - thanks! It seems like this is an edge case that we didn't test, apparently! Namely: having the \r\n sequence at the end of the SQL.

I'm not sure if your suggestion is the cleanest way to solve this issue, so I will probably look into this as well. Note that changes related to tokenization must be made for both the Python and the Rust tokenizer.

[georgesittas]

Check out the linked PR @mr-miles.