Support Chinese? about keybert

[MaartenGr]

Yes! Simply choose an embedding model that supports Chinese and you are good to go 😄

[marijndew]

With Chinese I think it doesn't work "out of the box." I suspect the model needs spaces between words which Chinese doesn't have. It worked for me after I used spaCy to segment the Chinese into space-separated words. Not sure if that's the proper way to go about it tho.

[MaartenGr]

Ah right, I forgot about the tokenizer. You need to make sure you use a Tokenizer in KeyBERT that supports tokenization of Chinese. I suggest installing jieba for this:

from sklearn.feature_extraction.text import CountVectorizer
import jieba

def tokenize_zh(text):
    words = jieba.lcut(text)
    return words

vectorizer = CountVectorizer(tokenizer=tokenize_zh)

Then, simply pass the vectorizer to your KeyBERT instance:

from keybert import KeyBERT

kw_model = KeyBERT()
keywords = kw_model.extract_keywords(doc, vectorizer=vectorizer)

[MaartenGr]

I will close this for now seeing as there hasn't been an update. However, feel free to re-open it if you are still experiencing this issue!