Support for chinese? about sumeval

[xiongma]

@icoxfog417 me too

[icoxfog417]

I want to support the Chinese! The spaCy can tokenize Chinese, so we think we can support Chinese like English manner.

And we need the following material to test behavior.

• Stop word/Stemming word list.
• regex to replace symbols.

@micxyj , @policeme Do you have any resource that would be helpful to implement the above two?

[xiongma]

I just have vocab stop list, Is the regex, just regex chinese or other word?

[micxyj]

• For Chinese stop words, you can refer this link: https://github.com/micxyj/extract_keywords/blob/master/stopwords.txt

• For regex, do you mean that you will replace Chinese punctuation like "，！。", etc?

[icoxfog417]

Thank you for offering stop word list!

For regex, do you mean that you will replace Chinese punctuation like "，！。", etc?

Yes. We have to exclude characters that have to be ignored when calculating ROUGE/BLEU.

[micxyj]

import re

example = "我：》？喜。，欢！@打~···篮「」球【】"
reg = re.compile(r'[\s+\.\!\/_,$%^*(+\“\’\”\'\"]+|[+——！，。：？、~@#￥%……&*（）【】「」《》·]+')
result = re.sub(reg, '', example)
print(result)

Can it works?

[xiongma]

Now, it support chinese? @icoxfog417

[icoxfog417]

I'm now tackling. Please wait a little.

[xiongma]

ok, thx your working!

[icoxfog417]

I implemented Chinese support. And now we need test summaries to test it.
Making test summaries requires careful work. It is needed to adjust the number of tokens.

• number of tokens of summaries.
• number of tokens of references.
• number of match tokens between summaries and references.

Here is Japanese sample (Each tokens is splited by space). So it will take a little time.

[icoxfog417]

Add test case by referring Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency.

[icoxfog417]

Merged #10. Please open the issue if you have some trouble with Chinese!