akirakubo / bert-japanese-aozora Goto Github PK

Hello, I'm the owner of awesome-bert-japanese, a list of pretrained BERT models available online.
@kkadowa taught me about your BERT model in the PR.
I'm really interested in adding your model to our list.

If you don't mind asking you a question, and adding your model to awesome-bert-japanese,
could you please tell me how to combine different morphological analyzers in preprocessing?

My questions here:

• himkt/awesome-bert-japanese#2 (comment)
• himkt/awesome-bert-japanese#2 (comment)

1. How to combine two analyzers (MeCab+unidic-cwj and MeCab+Unidic-qkana)
2. How to combine two analyzers (SudachiPy+SudachiDict_core-20191224 and MeCab+UniDic-qkana_1603)
3. In SudachiPy, what tokenization mode did you use? (ref, Multi-granular Tokenization in the official documentation)

I'd really appreciate your response.
Thanks!

Thanks for sharing models.

MeCab is BSD license.
Sudachi is Apache license.
unidic-cwj-2.3.0 is BSD license.

But unidic-qkana_1603 is "CC BY-NC-SA 4.0" license.

So I guess all models are "CC BY-NC-SA 4.0" license?