Comments (3)
The behavior of lemmatizer is as expected. Lemmatizer modifies suffix or prefix of a word. It is not able to modify word somewhere in the middle. Compound words (popular in germanic languages) also are lemmatized only by suffix.
However the problem may be related to tokenization of words containing - or '
If 'portes-fenêtre' would be tokenized to 'portes' '-' 'fenêtre' (splitted at hyphen) then individual words should be pos-tagged and lemmatized.
In the issue related to tokenization of 't-shirt' I made a fix forcing words to be splitted at hyphen. This is global change, not only for english language but for all languages. Hopefully it will fix also problems above.
I suspect that words containing ' (apostroph character) may also be inconsistently annotated so they are sometimes splitted, sometimes not. It may be next candidate to fix.
from babzel.
The new end-of-February model fixes all of these! Especially amazing is that "des chefs-d'oeuvre" is now lemmatized as "un", "chef", "-", "de", "œuvre". Note, this took care of handling of "d'" automagically!
from babzel.
I'm glad that small change, corrected the problem for many languages. Thank you for checking the models.
from babzel.
Related Issues (6)
- How to use the model from Java (not from Solr) ? HOT 8
- Add "end of sentence" characters specific for a language HOT 1
- Implement common (single) text preprocessor instead of two different preprocessors
- Inconsistent tokenization of "T-shirt". HOT 8
- Is the Chinese model for simplified han script or traditional? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from babzel.