miznokruge / segphrase-multilingual

SegPhrase working on Chinese and Arabic

SegPhrase

This repository entends English SegPhrase to Chinese (both traditional and simplified).

The underlying idea is to reuse the English implementation by adding an encoding/decoding layer in training and parsing process, respectively. During the training phase, we create a dictionary between Chinese word segments and English letters, encode the input data to English pseudo words. In the parsing phase, after identifying quality phrases in encoded new documents, we do the decoding to recover the readable Chinese.

Requirements

• g++ 4.8
• python 2.7
• pypy (used for efficiency consideration, set PYPY=python in *.sh scripts if pypy is not installed)
• scikit-learn (python package)
• jieba (python package)
• opencc 1.0.*

Build

SegPhrase can be easily built by Makefile in the terminal.

$ make

Default Run

$ ./train.sh  #train a segmenter and output phrase list as results/salient.csv.natural
$ ./parse.sh  #use the segmenter to parse new documents

Parameters - train.sh

RAW_TEXT=data/wiki.cleaned.txt

RAW_TEXT is the input of SegPhrase, where each line is a single document.

PHRASE_LIST=results/salient.csv.natural

PHRASE_LIST contains a list of Chinese phrases ranked by their predicted quality.

AUTO_LABEL=1
DATA_LABEL=data/wiki.label.auto

When AUTO_LABEL is set to 1, SegPhrase will automatically generate labels and save it to DATA_LABEL. Otherwise, it will load labels from DATA_LABEL.

KNOWLEDGE_BASE=data/wiki_labels_quality.txt
KNOWLEDGE_BASE_LARGE=data/wiki_labels_all.txt

We have two knowledge bases, the smaller one contains high quality phrases for positive labels while the larger one is used to exclude medium quality phrases for negative labels.

SUPPORT_THRESHOLD=10

A hard threshold of raw frequency is specified for frequent phrase mining, which will generate a candidate set.

OMP_NUM_THREADS=4

You can also specify how many threads can be used for SegPhrase

DISCARD_RATIO=0.00

The discard ratio (between 0 and 1) controls how many positive labels can be broken. It is typically small, for example, 0.00, 0.05, or 0.10. It should be EXACTLY 2 digits after decimal point.

MAX_ITERATION=5

This is the number of iterations of Viterbi training.

TOP_K=5

Uses top k segmentation solutions for each sentence in Viterbi training.

ALPHA=0.85

Alpha is used in the label propagation from phrases to unigrams.

Parameters - parse.sh

./bin/segphrase_parser results/segmentation.model results/salient.csv 5000 ./data/test.txt ./results/parsed.txt 0 ${TOP_K} ${OFFSET}

The first parameter is the segmentation model, which we saved in training process. The second parameter is the high quality phrases ranking list (together with unigrams). The third one determines how many top ranked phrases (unigrams) will be considered in this run of segmentation. This parameter should be dataset and application specific. The later two are the input and the output of corpus. The last one is a debug flag and you can just leave it as 0.

src/online_query/decoding.py

This script decodes the English letters back to Chinese characters. Phrase offsets in the document are also recorded.