Comments (3)
Thank you for this!
I agree that UniDic is the best choice, however, bundling the dictionary by default would most likely be an unwelcome change for most users. There is no need to host UniDic dictionaries in / by Memento, as they're actively maintained by NINJAL. Perhaps there could be a button to fetch the dictionary from NINJAL's site, if the user wishes to do so.
How MeCab works
Just to make sure that everybody reading this issue is on the same page regarding how MeCab works, I will try to concisely explain it:
Mecab segments a text based on the inflections, stems and word forms provided by a dictionary. This means that different dictionaries can segment differently. Additionally, since natural languages aren't perfectly rule based, a text string can have multiple, grammatically correct segmentations. MeCab segments every possible permutation and only returns the first one by default. The number of segmentations returned can be controlled with the -N[X]
flag. E.g.: -N3
to return the first 3 results. MeCab only truncates, so returning more results incurs 0 performance cost.
Different dictionaries structure their output differently, so whatever Memento implements will have to be dictionary specific. Ipadic is by far the worst one available. Jumandic works most of the time out of the box, Unidic can be made to work every time, but it is currently not a drop in replacement.
Let's look at 2 examples with 感じたんだ
:
Jumandic
echo 感じたんだ | mecab -N2 --dicdir /usr/lib64/mecab/dic/jumandic
感じた 動詞,*,ザ変動詞,タ形,感ずる,かんじた,代表表記:感ずる
んだ 助動詞,*,ナ形容詞,基本形,んだ,んだ,*
EOS
感じた 動詞,*,母音動詞,タ形,感じる,かんじた,補文ト 代表表記:感じる
んだ 助動詞,*,ナ形容詞,基本形,んだ,んだ,*
EOS
The first result from Jumandic returns 感ずる, the second returns 感じる. It is obviously possible to find both of these forms in JMdict just by looking up the stem, but that is currently besides the point. Accounting for multiple results from MeCab at index 4 gives us the verbs, along with the inflections.
UniDic
echo 感じたんだ | mecab -N2 --dicdir /usr/lib64/mecab/dic/unidic_spoken
感じ 動詞,一般,*,*,サ行変格,連用形-一般,カンズル,感ずる,感じ,カンジ,感ずる,カンズル,混,*,*,*,*,*,*,用,カンジ,カンズル,カンジ,カンズル,0,C2,M4@1,2106681492382337,7664
た 助動詞,*,*,*,助動詞-タ,連体形-一般,タ,た,た,タ,た,タ,和,*,*,*,*,*,*,助動,タ,タ,タ,タ,*,"動詞%F2@1,形容詞%F4@-2",*,5948916285711041,21642
ん 助詞,準体助詞,*,*,*,*,ノ,の,ん,ン,ん,ン,和,*,*,*,*,*,*,準助,ン,ン,ン,ン,*,"動詞%F2@0,形容詞%F2@-1",*,7968727735869952,28990
だ 助動詞,*,*,*,助動詞-ダ,終止形-一般,ダ,だ,だ,ダ,だ,ダ,和,*,*,*,*,*,*,助動,ダ,ダ,ダ,ダ,*,名詞%F1,*,6299110739157675,22916
EOS
感じ 動詞,一般,*,*,上一段-ザ行,連用形-一般,カンズル,感ずる,感じ,カンジ,感じる,カンジル,混,*,*,*,*,*,*,用,カンジ,カンジル,カンジ,カンジル,0,C2,M4@1,2106672902447745,7664
た 助動詞,*,*,*,助動詞-タ,連体形-一般,タ,た,た,タ,た,タ,和,*,*,*,*,*,*,助動,タ,タ,タ,タ,*,"動詞%F2@1,形容詞%F4@-2",*,5948916285711041,21642
ん 助詞,準体助詞,*,*,*,*,ノ,の,ん,ン,ん,ン,和,*,*,*,*,*,*,準助,ン,ン,ン,ン,*,"動詞%F2@0,形容詞%F2@-1",*,7968727735869952,28990
だ 助動詞,*,*,*,助動詞-ダ,終止形-一般,ダ,だ,だ,ダ,だ,ダ,和,*,*,*,*,*,*,助動,ダ,ダ,ダ,ダ,*,名詞%F1,*,6299110739157675,22916
EOS
UniDic seems overly verbose and scary at first, but it is very well documented. Columns of particular interest are index 7 and 10. 語彙素 and 書字形基本形 respectively. Once again, while it would be possible to find 感じる in JMdict from the first result alone, but the second result gives it directly, with the appropriate inflections.
UniDic has formatting issues that currently crashes Memento
This was possibly solved sometime in the past, because setting UniDic as my system dictionary and changing:
#define WORD_INDEX 6
to
#define WORD_INDEX 7
in dictionary.cpp
line 269
makes Memento use UniDic, and it works mostly correctly. (Although as I mentioned earlier, since it only looks at column 7, it only finds 感ずる and not 感じる. Important to note that this is not a shortcoming of Mecab, it is simply not fully implemented yet.
With the current MeCab implementation, Jumandic with #define WORD_INDEX 4
is a drop in upgrade to ipadic in every way, since it is a more complete dictionary. Even though Unidic is far more extensive, the way it orders the results happens to be less compatible with the current MeCab implementation in Memento.
However, for a proper grammar implementation, I suggest basing it on Unidic's format, as it has numerous dictionaries for both modern and classical Japanese.
My C++ knowledge is somewhat limited, and I'm almost entirely unfamiliar with how Memento currently works, but I would gladly help with parsing the output.
EDIT:
I can confirm that with your example subtitle of
〝民間南極観測隊
3年ぶりに派遣決まる〞
and Unidic, Memento does indeed crash. Removing 〝
from the subtitles fixed this issue. I'm not entirely sure why it happens, as Unidic itself handles 〝
just fine. However, sanitising the subtitles is a reasonable solution.
from memento.
I'm not against it, but there's no way I'm bundling UniDic with Memento due to the size. I doubt I'll be changing away from ipadic either. This is because every distro I know of bundles ipadic as the default MeCab dictionary. Since dictionary formats can't be handled the same way across dictionaries, the safest option is to use ipadic as it has already proven itself as reliable.
That said, I'm not against having an option to use UniDic over ipadic. The implementation details of these are something that I'm more interested in discussing in #210 than here. It would definitely involve having the user download UniDic after the fact in some fashion.
from memento.
It would definitely involve having the user download UniDic after the fact in some fashion
I agree. Even the current yt-dlp on Mac kind of workaround would be appreciated.
from memento.
Related Issues (20)
- Error adding term Operation canceled HOT 12
- Subtitle search only works when media is paused HOT 3
- [Feature request] Export only lowest frequency value HOT 1
- Memento fills empty fields with html HOT 1
- Play audio doesn't work HOT 1
- Can't open on MAC M3 Arm HOT 6
- what settings are needed for YouTube links to work? HOT 4
- Kanji Stroke Order font HOT 1
- [Feature Request] Glossary Selection Marker HOT 12
- [Feature Request] Subtitle Delay HOT 3
- QT6 migration causes subtitle position to shift when on screen controller is shown/hidden HOT 1
- Lack of Styling on Anki Cards Added via Memento HOT 4
- jp mining note support HOT 2
- Setting secondary_sub_visibility to 'auto' within the mpvacious plugin does not work HOT 2
- Any configuration guide? HOT 1
- Jellyfin mpv shim? HOT 16
- Video does not play HOT 4
- Improve popup dictionary styling HOT 5
- Partially visible video when using display scaling HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from memento.