6gsn / marine

MARINE : Multi-task leaRnIng-based JapaNese accent Estimation

marine is a tool kit for building the Japanese accent estimation model proposed in our paper (demo).

For academic use, please cite the following paper (ISCA archive).

@inproceedings{park22b_interspeech,
  author={Byeongseon Park and Ryuichi Yamamoto and Kentaro Tachibana},
  title={{A Unified Accent Estimation Method Based on Multi-Task Learning for Japanese Text-to-Speech}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={1931--1935},
  doi={10.21437/Interspeech.2022-334}
}

Notice

The model included in this package is trained using JSUT corpus, which is not the same as the dataset in our paper. Therefore, the model's performance is also not equal to the performance introduced in our paper.

Get started

Installation

$ pip install marine

For development

$ pip install -e ".[dev]"

Quick demo

In [1]: from marine.predict import Predictor

In [2]: nodes = [{"surface": "こんにちは", "pos": "感動詞:*:*:*", "pron": "コンニチワ", "c_type": "*", "c_form": "*", "accent_type": 0, "accent_con_type": "-1", "chain_flag": -1}]

In [3]: predictor = Predictor()

In [4]: predictor.predict([nodes])
Out[4]:
{'mora': [['コ', 'ン', 'ニ', 'チ', 'ワ']],
 'intonation_phrase_boundary': [[0, 0, 0, 0, 0]],
 'accent_phrase_boundary': [[0, 0, 0, 0, 0]],
 'accent_status': [[0, 0, 0, 0, 0]]}

In [5]: predictor.predict([nodes], accent_represent_mode="high_low")
Out[5]:
{'mora': [['コ', 'ン', 'ニ', 'チ', 'ワ']],
 'intonation_phrase_boundary': [[0, 0, 0, 0, 0]],
 'accent_phrase_boundary': [[0, 0, 0, 0, 0]],
 'accent_status': [[0, 1, 1, 1, 1]]}

Build model yourself

Coming soon...

LICENSE

• marine: Apache 2.0 license (LICENSE)
• JSUT: CC-BY-SA 4.0 license, etc. (Please check jsut-label/LICENCE.txt)

marine's Issues

Add sample recipes

Ref. #17

Currently, the sample model has a different architecture than the model proposed in our paper.
So, we should explain the background (i.e., why we did like that) and add the recipes for other architectures with tips.

Add docs

Currently, there are no tips for model customizing and other development issues.
We should build the document for this repository as soon as possible ⏳

Could you kindly share how to train new model?

Hi!
I appreciate for sharing your marine. I used your pretrained model trained by JSUT.
Though pretrained model works very well in ordinary corpus, when I used it in my original corpus (corpus about games or animations), its accuracy was about 50%.
So I wanted to build my model by using my corpus set and testify accuracy.

I know you described it will come soon. I'm very looking forward to it, and want to know when you plan to share it.
Thank you.

pretrain model license?

This repository itself seems to be Apache.
But when I use this it claims non-commercial.

marine/marine/utils/pretrained.py

Line 60 in 3ad2689

"The use of pre-trained models is permitted for non-commercial use only."

Does this mean you trained model by non commercial data?

Which dictionary does the pretrained model use?

Hi, thanks for sharing such a great work!

As shown in your paper, unidic is used in pre-processing. But the README in this repo says the provided model is not the same as the one in your paper. And I notice that the provided model can work with OpenJTalk, which is based on naist-jdic.

So I wonder which dictionary the provided model uses?