Comments (13)
How about this?
https://github.com/knowitall/morpha
from bpe_analysis.
Should we keep the plus sign?
In my Indonesian morphology normalizer script I didn't. Also, are you working on this?
from bpe_analysis.
Should we keep the plus sign?
No, +
is just for an illustration purpose.
I will do English and Japanese normalization. Were you also working on this?
from bpe_analysis.
Yes, the scripts for Indonesian is at https://github.com/justhalf/bpe_analysis/blob/master/morphind/process_txt.py
It uses MorphInd for the morphology analyzer.
No,
+
is just for an illustration purpose.
I asked because in the Indonesian one I explicitly remove +
. I think we should remove the plus sign, yeah.
from bpe_analysis.
Did you start English and Japanese normalization, too?
from bpe_analysis.
Did you start English and Japanese normalization, too?
No, I haven't started. I didn't know which morphology analyzer to use. But if we have them, we can simply replace the subprocess call with the corresponding call.
from bpe_analysis.
In this paper it says the lexicon (19MB) is large:
Does anyone know a good English morphology analyzer? I was surprised to find none, only Morfessor, which was automatic.
from bpe_analysis.
Maybe this one? http://wiki.apertium.org/wiki/Lttoolbox
from bpe_analysis.
Maybe this one? http://wiki.apertium.org/wiki/Lttoolbox
Does this output surface forms of morphemes?
This statistical morphological segmenter can generate normalized surface forms like un+test+able+ly
:
https://github.com/ryancotterell/treeseg
We can search for similar studies by "morphological segmentation" rather than "morphological analysis"
from bpe_analysis.
Based on my cursory look, it seems so.
This statistical morphological segmenter can generate normalized surface forms like un+test+able+ly:
https://github.com/ryancotterell/treeseg
That's a good one, since it is modern. I was looking at more that has more manual analysis, since it will be less automatic, e.g., FST. But couldn't find FST for English.
from bpe_analysis.
That looks good. You have one for Japanese as well? (I guess we don't need this for Chinese?)
from bpe_analysis.
Fortunately, Japanese segmentation by UDPipe is already morpheme segmentation and has normalized forms. I think we don't need normalization for Chinese.
Can you do English normalization?
from bpe_analysis.
I am trying. Morpha apparently only handles plural nouns and verb inflections, but not derivations. So happiness
stays as is.
from bpe_analysis.
Related Issues (7)
- Statistics of the Corpus HOT 10
- BPE training status on WIKI-data HOT 19
- Analysis on affixes HOT 1
- Discussion for the final report HOT 18
- LM-scores for UD-En HOT 3
- Slides Updated HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bpe_analysis.