[BUG] Fast tokenizer does not deal with AddedTokens properly(no problem in Transformers python tokenizer impl.) about tokenizers

[MilkClouds]

$ pip list | grep tokenizer                                                                                                                                                                   ✹
tokenizers                0.19.1
$ pip list | grep transformers                                                                                                                                                               
sentence-transformers     3.0.0
transformers              4.41.2

[ArthurZucker]

Hey! I think most of these can be removed if you set the legacy=False flag when initializing the tokenizer. I'll talk to the M4 team about this.

Basically the normalizer is prepending a space before each token, and before each split! For more details huggingface/transformers#28881