Some common used python tool for text pre-processing and NLP tools, including chinese and english.
- jibea :
pip install jieba
- opencc for python, download from link
- NLTK :
pip install nltk
- gensim :
pip install gensim
-
Chinese segment
We use jieba to segment chinese corpus.
Commandpython ChineseSegment.py input.txt output.txt user_dict(default is None)
to segment a corpus file or reference sample code in your code.# sample code # init jieba segment = ChineseSegment() # segment = ChineseSegment('data//dictionary//user_dict.dict') arr = ['測試','正在測試','今天晚餐吃啥','小明碩士畢業於中國科學院計算所'] # segment chinese array or string print(segment.cut(arr[3])) print(segment.cut(arr)) # segment chinese array or string for search mode print(segment.cutForSearch(arr[3])) # segment and POStagging input sentence print(segment.tokenizer(arr[3]))
-
Conversion in traditional chinese and simplified chinese We use opencc in python to convert.
Commandpython opencc.py input.txt output.txt conversion_model(default=s2t)
to convert a corpus file -
Common corpus pre-processing This tool contain stemming, stopwords, removing punctuation, removing url and convert word to lower case.
We adoptargparse
to execute command. You need input-i
and-o
to assign the path of input file and output file. Then, you can choose-s
,-p
,-u
and-sw
to stemming, filter punctuation, filter url and filter stopwords. If you want to turn on all settings, you can use-a
parameter.
For example,python Preprocessing.py -i input.txt -o output.txt -a
.
We also provide sample code(demo function) inPreprocessing.py
.
- Word2Vec
Just a example of how to use wWrd2Vec ingensim
. Word2Vec used to find similar opinion word and word clustering. You can reference the code ofword2vec.py
to train a new model and test it. I reference this website. If you need a pre-trained model for english, you can use the model trained by google. - Doc2Vec
Just a example of how to use Dord2Vec ingensim
.
Doc2Vec used to implement document clustering, support QA sustem, or find similar opinion sentence. You can reference the code ofdoc2vec.py
to train a new model and test it. I reference this website.
- More preprocessing tool, like parse tree, basic statistics, etc.
- Some example of NLP tools, like topic model, named entity recognition, etc.