Multibyte characters break positions about javascript-driver

[creachadair]

See also bblfsh/documentation#228, which was motivated by issues like this. I think we will need to make a clearer contract about what constitutes "acceptable" changes to the UAST structure as a result of bug fixes and natural evolution

[EgorBu]

One more file: probably because of 💾

[dennwc]

I cannot reproduce this issue.

According to the comment on the Offset field:

Offset is ... an absolute byte offset.

@EgorBu Maybe you are using the offset as a UTF8 character offset instead of a byte offset?

[dennwc]

OK, nevermind, I reproduced an issue. The native driver returns JS offsets in UTF8 runes, not bytes.

[zurk]

@dennwc do you plan to fix it not only in the lastest v2 driver but for the latest v1 too?
yeah, we still use old drivers, sorry for that, but as soon as we have enough time we plan to switch to v2.

[dennwc]

I don't think we can backport it down to v1, but all new drivers should be able to run in v1 compatibility mode.

[zurk]

oh, I did not know it is possible. Sorry for offtopic, but how can I enable it? From my experiments, I remember that pure driver replacement gives different UAST structure and it breaks ML team code.

[zurk]

@dennwc what about v1 compatibility mode and my last question?

[dennwc]

@zurk Yeah, sorry, missed the notification.

I checked the code, v1 compatibility is still enabled in the new bblfshd releases. So you can use new driver to get the good old Node structure.

But, since we fixed lots of things in new drivers, you might see some slight changes in the AST. Those are mostly bug fixes, but I don't know if they are severe enough to break assumptions in your pipeline.

I would say you should give it a try and feel free to ping me if you find any differences that break the pipeline. Some might be fixable upstream.

[zurk]

ok, thank you, @dennwc! I will check it.