daniellin94144 / dual-textless-sqa

Textless (ASR-transcript free) Spoken Question Answering. The official release of NMSQA dataset and the implementation of "DUAL: Textless Spoken Question Answering with Speech Discrete Unit Adaptive Learning" paper.

DUAL-textless-SQA

This repository is the official implementation for DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering paper, and the release of the Natural Multi-speakers Spoken Question Answering (NMSQA) dataset.

Installation

Model

• [Pretrained DUAL model on HuBERT-128]

Dataset

Download the NMSQA dataset

• [Original dataset]
• [Parquet Format dataset]
• [Huggingface Format dataset]

Data Preparation for Original Dataset

Preprocessed data link (including passage merging and unit-level labels, updated with question code): [link]

• Directory format

  • train
  • dev
  • test

• Files

  • For train and dev split {split}-answer-span.csv: answer time span in seconds meta-{split}.csv: the duration, speaker, and transcription of each utterance {split}-textgrid.tar.gz: force alignment of each utterance {split}_audio.tar.gz: utterance waveform files {split}_hash2question.json: map the hash value to question id
  • For test split lxt_sqa.tar.gz: contains all audio files in audio and transcriptions meta-lxt.csv: the duration, speaker, and transcription of each utterance test/test-SQuAD/test-SQuAD-answer-span.csv: the answer span in the test-SQuAD split test/test-OOD/test-OOD-answer-span.csv: the answer span in the test-OOD split

  NOTE Current the spoken passage is split to segments of utterances. For the standard QA task, you should merge the segments back to the whole passages. The suffix of -1, -2, ..., -n is the segment number of specific passage.

  • Speech Content Encoder Please see details in speeech-content-encoder.
  • Pre-process the QA labels

  python code_answer.py
  

Parquet Format & Huggingface Format dataset

It basically follow the same file format as the Origin SQuAD with the following extra field:

{
   "id": Same as SQuAD,
   "title": Same as SQuAD,
   "context": Same as SQuAD,
   "question": Same as SQuAD,
   "answers":{
      "answer_start": Same as SQuAD,
      "audio_full_answer_end":[], Audio answer end position in second
      "audio_full_answer_start":[], Audio answer start position in second
      "audio_full_neg_answer_end":[], Audio answer end position in second that using the same words but not the correct one
      "audio_full_neg_answer_start":[], Audio answer start position in second that using the same words but not the correct one
      "audio_segment_answer_end":[],
      "audio_segment_answer_start":[],
      "text": Same as SQuAD
   },
   "content_segment_audio_path": Segment Audio Path,
   "content_full_audio_path": Complete Audio Path,
   "content_audio_sampling_rate": Audio Sampling Rate,
   "content_audio_speaker": Audio Speaker,
   "content_segment_text":"",
   "content_segment_normalized_text": Normalized Text for generating audio,
   "question_audio_path": Question Audio Path,
   "question_audio_sampling_rate": Audio Sampling Rate,
   "question_audio_speaker": Audio Speaker,
   "question_normalized_text": Normalized Text for generating audio,
}

Training

python train.py --exp_name [exp name] --config baseline.yaml

Evaluation

python evaluate.py --data_dir [data dir path] --model_path [model checkpoint dir] --output_dir [output dir path] --out_fname [output name]

Results

Discrete unit	PLM	dev FF1	dev AOS	test FF1	test AOS
HuBERT-64	Longformer	47.8	42.4	39.0	33.0
HuBERT-128	Longformer	54.2	48.5	56.0	49.1
HuBERT-512	Longformer	55.0	49.6	17.3	12.5

Contact

Guan-Ting Lin (Email: [email protected]) Eric Lam (Email: [email protected])

Citation

@inproceedings{lin22c_interspeech,
  author={Guan-Ting Lin and Yung-Sung Chuang and Ho-Lam Chung and Shu-wen Yang and Hsuan-Jui Chen and Shuyan Annie Dong and Shang-Wen Li and Abdelrahman Mohamed and Hung-yi Lee and Lin-shan Lee},
  title={{DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={5165--5169},
  doi={10.21437/Interspeech.2022-612}
}

dual-textless-sqa's Issues

Preprocessed units for the segments

Hello!

I'm recently doing some experiments on NMSQA, and the code for DUAL provided here are really helpful! While I encountered some difficulty building the units using scripts provided to reproduce the results. Particularly, I'm trying to extract the units for each segment of context, while the preprocessed ones currently provided in the repo are already concatenated for each article following the standard QA scheme (using the merge_passage.py, I guess). May I know if the preprocessed units for each segment could be provided?

Thank you!

P.S. Just seen you and had some chat on Interspeech at the poster. The work was really impressive and useful for us :)

no question code available in processed data

Brilliant works! We are trying to train this model and we find that only codes of contexts are provided in the processed data link. Would you mind providing the processed codes of questions of train and dev split? since the preprocess pipeline is a little time consuming.