confused about performance in chinese text about keybert

[MaartenGr]

You need to make sure you use a Tokenizer in KeyBERT that supports tokenization of Chinese. I suggest installing jieba for this:

from sklearn.feature_extraction.text import CountVectorizer
import jieba

def tokenize_zh(text):
    words = jieba.lcut(text)
    return words

vectorizer = CountVectorizer(tokenizer=tokenize_zh)

Then, simply pass the vectorizer to your KeyBERT instance:

from keybert import KeyBERT

kw_model = KeyBERT()
keywords = kw_model.extract_keywords(doc, vectorizer=vectorizer)

[rattlesnakey]

the output like the picture above

[rattlesnakey]

Thx so much!!! | | hengyuan_blcu | | ***@***.*** | 签名由网易邮箱大师定制 On 09/28/2021 02:42，Maarten ***@***.***> wrote： You need to make sure you use a Tokenizer in KeyBERT that supports tokenization of Chinese. I suggest installing jieba for this: fromsklearn.feature_extraction.textimportCountVectorizerimportjiebadeftokenize_zh(text): words=jieba.lcut(text) returnwordsvectorizer=CountVectorizer(tokenizer=tokenize_zh) Then, simply pass the vectorizer to your KeyBERT instance: fromkeybertimportKeyBERTkw_model=KeyBERT() keywords=kw_model.extract_keywords(doc, vectorizer=vectorizer) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.