LLama.Native.SafeLLamaContextHandle.Tokenize bug? about llamasharp

[dluc]

@martindevans I looked at the fix, and if I understand correctly \n is 3 tokens, could you confirm?

Until the bugfix is released, I'll hard code this edge case in Kernel Memory (which is using 0.9.1) to avoid exceptions.

[aletfa]

https://github.com/aletfa/doctalk @martindevans TY.

[martindevans]

Thanks for reporting this, I've reproduced it in a test and made some adjustments to the tokenizer which should fix it.

[dluc]

Btw in 0.9.1 the exception occurs also with these strings:

• \t
• \t\t
• \t\t\t
• ......
• \n\n
• \n\n\n
• \n\n\n\n
• ....
• \t\n
• \t\n\t\n\n\n\n\t\t
• ....
• \b
• \v
• \0

[martindevans]

Thanks for those examples, I've just added a unit test covering them (it passes, so it looks like the fix works).

if I understand correctly \n is 3 tokens

It's actually:

• 1: Beginning of string token
• 29871: A space (llama.cpp tokenizer really likes to add a space at the start)
• 13: The actual newline token

[vvdb-architecture]

Another one is \nk.

[martindevans]

All of these should be fixed by the changes currently in master :)

[aletfa]

When a new Nuget package will be released with these fixes?

[martindevans]

Very soon, it's been held up by some other OpenCL related changes. I hope by the end of this week.

[aletfa]

I confirm that the fix resolves all the problems. TY. I'll wait for the next week.

[martindevans]

I jsut released 0.10.0 with these changes :)