Comments (8)
I tested with other compound words. It seems in majority case, "-" is treated as a separate token. But there are cases where the compound words are treated as one (for example "e-commerce", "non-profilt", "multi-lingual"), and there are compound words where the hyphen is attached to the second word ("turn-key" => ["turn", "-key"]; "word-for-word" => ["word", "-", "for", "-word"]; "world-class" => ["world", "-class"]).
I suspect this might be caused by inconsistency in the annotation.
from babzel.
You made an interesting observation!
Your hypothesis that hyphen words are inconsistently annotated seems right.
I looked in *.conllu file and hyphen words are most often splited but there are cases that they are treated as single word.
I noticed examples: so-called, african-american, avant-garde, run-on and others.
It may be the cause of tokenizer inconsistent behavior.
from babzel.
I think it can be fixed. During implementation I noticed inconsistent annotations for 'de', 'fi' language and implemented so-called Transformer. Transformer read conllu sentences and tries to fix it or reject it.
EnTransformer can check sentences and if some word is not splitted hyphen word then reject such sentence. Tokenizer model should then always split a word at hyphen.
from babzel.
That sounds very promising! Will it take a while to get the model fixed?
from babzel.
Models are automatically recomputed every month on 27th day. However I don't have much time recently, so I am not sure if I make a fix soon.
from babzel.
I've just make a small fix. Sentences which contains not splitted words with hyphens are rejected. Hopefully, trained models will consistently split words at hyphens.
Please check after 27-th february when new models are recomputed.
from babzel.
Thank you. I'll check out the new model when it becomes available!
from babzel.
In my smoke test, the new model handled compound words with "-" very consistently. "yellow t-shirt!" is now tokenized as "yellow", "t", "-", "shirt" and "!". "post-brexit", is now "post", "-", and "brexit". It seems this model fixed the issue. Thank you!
from babzel.
Related Issues (6)
- How to use the model from Java (not from Solr) ? HOT 8
- Add "end of sentence" characters specific for a language HOT 1
- Implement common (single) text preprocessor instead of two different preprocessors
- Is the Chinese model for simplified han script or traditional? HOT 1
- Tokenization and lemmatization behavior of French hyphened compound words HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from babzel.