rhss10 / joint-apa-mdd-mtl

Code for the Interspeech 2023 paper "A Joint Model for Pronunciation Assessment and Mispronunciation Detection and Diagnosis with Multi-task Learning"

Thanks for visiting my GitHub page! 👋

1. Interests: Automatic speech recognition, non-native speech processing, and speech processing/AI in general

2. Skills: Python, PyTorch, Transformers, Fairseq, Kaldi, Docker, C, Java, Linux

3. Education

• B.A. in Linguistics / B.E. in Computer Science and Engineering at Seoul National University (2016-2021)
• M.A. in Linguistics at Seoul National University (2021-2023)

4. Experiences

• Exchange student at the University of Geneva (Spring of 2019)
• Intern at Speech Synthesis Team, Speech AI Lab, AI Center, NCSOFT (20.07-20.08, 20.11-21.02)
• Associate at AI Unit, Digital Innovation Division, Shinhan Bank (23.08 ~)

5. Publications

• Yeo, E. J.*, Ryu, H., Lee, J., Kim, S., & Chung, M. (2023). Comparison of L2 Korean Pronunciation Error Patterns from Five L1 Backgrounds by using Automatic Phonetic Transcription. Proc. ICPhS 2023. (https://arxiv.org/abs/2306.10821)
• Ryu, H., Kim, S., & Chung, M. (2023). A Joint Model for Pronunciation Assessment and Mispronunciation Detection and Diagnosis with Multi-task Learning. Proc. Interspeech 2023. (https://www.isca-speech.org/archive/pdfs/interspeech_2023/ryu23_interspeech.pdf)

6. Projects

• Japanese ASR (National Information Society Agency (NIA) 2022-2023)
• L2 Japanese phone recognition of Korean learners (National Information Society Agency (NIA) 2022-2023)
• L2 Korean pronunciation assessment of non-native learners of multinational backgrounds (National Information Society Agency (NIA) 2022-2023)

joint-apa-mdd-mtl's Issues

A question about training a joint model of APA and MDD with multi-task learning.

During the training of the Joint-CAPT-L1 model, both ctc_loss and cls_loss are NaN, and I’m not sure what the reason is. I followed your code and steps. I hope to get your answer.

which version of python should I use?