canclid / canto-filter Goto Github PK
View Code? Open in Web Editor NEW粵文語料篩選器 Cantonese text filter
Home Page: https://pypi.org/project/canto-filter/
License: MIT License
粵文語料篩選器 Cantonese text filter
Home Page: https://pypi.org/project/canto-filter/
License: MIT License
Because we don't have a reliable Cantonese tokenizer, all the phrases added to the classifier need to satisfy the following property, in addition to not being shared by Mandarin:
Frequency of the phrase is very low in an untokenized Mandarin corpus. We don't want incorrect word boudaries to mess up our classification. For example, if the phrases like 全家下棋 are common in Mandarin, we don't want to add the phrase 家下 to canto_unique
. The classifier will incorrectly estimate the word boundary in Mandarin texts and judge a Mandarin text to be Cantonese.
Words.hk provides lists of Cantonese phrases and characters as well as their common variants. It also has labels on whether the phrase is a 書面語. We first exclude any 書面語 words from our candidates for canto_unique
. Then, we can search each of the remaining characters or phrases in a large Mandarin corpus (e.g. Wudao Corpus 2.0 publushed a 200GB corpus for the public). If the character/phrase appears under a certain frequency, we add it to the canto_unique
list. Alternatively, we can generate such a list of candidates and manually review them before adding them to the list.
Since we don't have a Cantonese tokenizer, we have to rely on a coarser statistical estimation. It might result in some false positives (classified as Cantonese but is actually Mandarin) and some false negatives (the Cantonese word is disguised under 假借 or it's simply not on our list). Under these constraints, the best approach is to increase the number of canto_unique
words that are very rare in Mandarin. Another approach is to add more mando_unique words that appear very rarely in Cantonese. This would require us to lookup the words in a large Cantonese corpus (e.g. LIHKG forum). However, since the written Cantonese corpuses I'm aware of are generally much smaller than their Mandarin counterparts and represent a certain age-range, I think the former approach relying on Mandarin corpuses would yield a more reliable classification.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.