Loading `tokenizer.model` with Rust API about tokenizers

[ArthurZucker]

You cannot load a tokenizer.model, you need to write a converter.
This is because it does not come from the tokenizers library but from either tiktoken or sentencepiece and there is no secret recipe. We need to adapt to the content of the file, but this is not super straight forward.

https://github.com/huggingface/transformers/blob/main/src/transformers/convert_slow_tokenizer.py#L544 is the simplest way to understand the process!

[EricLBuehler]

Ok, I understand. Do you know of a way or a library to do this in Rust without reaching for the Python transformers converter?

[ArthurZucker]

A library no, but we should be able to come up with a small rust code to do this 😉

[EricLBuehler]

@ArthurZucker are there any specifications or example loaders which I can look at to implement this?

[chenwanqq]

I also have the same question, for llava reasons😉