a problem of Chinese coding about llamasharp

[martindevans]

Could you please paste the characters that you typed here as text? I can't investigate the problem otherwise since I can't type those characters!

[yuydev]

你好--你好。我今天能为?????????么? used the UTF-8 character set ;

[yuydev]

你好--你好。我今天能为?????????么? used the UTF-8 character set ;

[AsakusaRinne]

are u using Baichuan model?

[yuydev]

uesd the ggml-vic13b-q5_1.bin

[yuydev]

[Jiahui-Young]

hello，are u fix this problem now？I meet the same problem. thx~

[AsakusaRinne]

hello，are u fix this problem now？I meet the same problem. thx~

Some time ago I met a case that the response went wrong when Chinese and English mixed in the prompt but it was okay when using pure Chinese as prompt. Could you please change the prompt to have a try?

[martindevans]

This should be fixed in LLamaSharp 0.7.0 for the StatelessExecutor (only the stateless executor, the others need more work). Can you confirm if it's fixed for you?

[chenzanyu]

InteractiveExecutor，I have the same problem, is there a solution

[martindevans]

Unfortunately no, not yet. Someone needs to go in and modify the InteractiveExecutor to use the new StreamingTokenDecoder.

[AsakusaRinne]

I've located as the problem as GB2312 encoding. I'll fix it soon with a partial refactor of the executor with StreamingTokenDecoder as Martin mentioned. And I'll appreciate for any help about it (next week I'll have some busy days). :)

[martindevans]

PR #293 has just added the new decoder into the base executor. Can anyone here please pull the master branch and test if this is fixed with all executors now? Thanks.

[AsakusaRinne]

PR #293 has just added the new decoder into the base executor. Can anyone here please pull the master branch and test if this is fixed with all executors now? Thanks.

I've tested yesterday but unfortunately it still doesn't work for Chinese. Windows uses GB2312 encoding by default for Chinese and we can get correct string by adding GB2312 to System.Text.Encoding. However it seems the output of the model is totally a mess. I'm not sure it's because of the tokenizer or detokenizer but I prefer detokenizer to be blamed because the output are all meaningless Chinese, instead of other characters.

[martindevans]

Ah ok, I thought it was the multi-token-single-character problem, but I guess not :(

[owxy]

The same problem still exists now, as Chinese characters are decomposed and combined into strange symbols：FlagAlpha-Llama2-Chinese-13b-Chat.Q2_K.gguf

[AsakusaRinne]

Update: I've found a way to deal with Chinese decoding however still not robust. Please wait for a while and we'll include the fix in the next release. :)

[AsakusaRinne]

This problem has been fixed with #326 and an example has been added. Could you please take a try with the master branch? Note that in our test, we found some Chinese model such as baichuan has a strange behaviour while others don't. If you're not sure which model to use, please consider llama2-chinese-alpaca.

[AsakusaRinne]

Close this issue as completed, please feel free to reopen it if there's any problem.