rungjoo / compm

Context Modeling with Speaker's Pre-trained Memory Tracking for Emotion Recognition in Conversation (NAACL 2022)

CoMPM: Context Modeling with Speaker's Pre-trained Memory Tracking for Emotion Recognition in Conversation (NAACL 2022)

The overall flow of our model

Requirements

1. Pytorch 1.8
2. Python 3.6
3. Transformer 4.4.0
4. sklearn

Datasets

Each data is split into train/dev/test in the dataset folder.

1. IEMOCAP
2. DailyDialog
3. MELD
4. EmoryNLP

Train

For CoMPM, CoMPM(s), CoMPM(f)

In this code, the batch size = 1. We do not add padding when the batch is greater than 1.

Argument

• pretrained: type of model (CoM and PM) (default: roberta-large)
• initial: initial weights of the model (pretrained or scratch) (default: pretrained)
• cls: label class (emotion or sentiment) (default: emotion)
• dataset: one of 4 dataset (dailydialog, EMORY, iemocap, MELD)
• sample: ratio of the number of the train dataset (default: 1.0)
• freeze: Whether to learn the PM or not

python3 train.py --initial {pretrained or scratch} --cls {emotion or sentiment} --dataset {dataset} {--freeze}

For a combination of CoM and PM (based on different model)

Argument

• context_type: type of CoM
• speaker_type: type of PM

cd CoMPM_diff
python3 train.py {--argument}

For CoM or PM

cd CoM or PM
python3 train.py {--argument}

Testing with pretrained CoMPM

• Naver drive
• Unpack model.tar.gz and replace it in {dataset}_models/roberta-large/pretrained/no_freeze/{class}/{sampling}/model.bin
  • dataset: dailydialog, EMORY, iemocap, MELD
  • class: "emotion" or "sentiment"
  • sampling: 0.0 ~ 1.0, default: 1.0

python3 test.py

Test result for one seed. In the paper, the performance of CoMPM was reported as an average of three seeds.

Model	Dataset (emotion)	Performace: one seed (paper)
CoMPM	IEMOCAP	66.33 (66.33)
CoMPM	DailyDialog	52.46/60.41 (53.15/60.34)
CoMPM	MELD	65.53 (66.52)
CoMPM	EmoryNLP	38.56 (37.37)

Citation

@inproceedings{lee-lee-2022-compm,
    title = "{C}o{MPM}: Context Modeling with Speaker{'}s Pre-trained Memory Tracking for Emotion Recognition in Conversation",
    author = "Lee, Joosung  and
      Lee, Wooin",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.416",
    doi = "10.18653/v1/2022.naacl-main.416",
    pages = "5669--5679",
}

compm's Issues

A question about comparison experiment, why didn't you compare with CESTa on dailydialog dataset?

Contextualized Emotion Recognition in Conversation as Sequence Tagging seems to be one influential article in ERC area, I wonder that why didn't you compare with CESTa on dailydialog dataset? Is there any specific reason?

pytorch-cuda version for the source code?

Hi,

Do you know which version of cuda the code was built using? Prerequisites listed pytorch 1.8 but didn't say with which version of cuda. I am finding it very difficult to run because I couldn't find torch 1.8 anywhere. Appreciate your help!

Can I ignore this warning? UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.

I met the warning below:
/root/miniconda3/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1471:
UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))

But the accuracy still printed. And seems like the train can be continue. I am wondering will this affect the training?

reproduce your work on MELD dataset, the weight F1 score is 66.52

your work is interesting and perfect. don't introduce external knowledge, just dependent on CoM and PM.

when i try to reproduce same result in MELD dataset. I want to achieve same result F1 score is 66.52.

use the command :python test.py? it is right? do need to setup other experiments.

could you please give me a hand? many thanks

Can I use this work for sentiment analysis of dyadic conversations?

Hi,

I am working on sentiment analysis of dyadic conversations. Can I use your work for this purpose?

I understand that this work can help with utterance-level sentiment/emotion recognition in conversations? Now from there, if I want to recognize sentiment/emotion of the whole conversation, are you aware of any techniques that might help?

Thank you so much for your help!

Possible typo in the paper

The paper explains the CoMPM figure in Page 4 and states:
Figure 2: Our model consists of two modules: a context embedding module and a pre-trained memory module.
The figure shows an example of predicting emotion of u 6 , from a 6-turn dialogue context. A, B, and C refer to the
participants in the conversation, where s A = p u 1 = p u 3 = p u 6 , s B = p u 2 = p u 5 , s C = p u 3 . W o and W p are
linear matrices.
I believe s C = p u 3 should be s C = p u 4

I wonder if the model works fine when batch is not 1?

I don't see any operations for attention_mask, which means if the Roberta model will set all attention_mask to 1?

Unexpected key(s) in state_dict: "context_model.embeddings.position_ids", "speaker_model.embeddings.position_ids"

Hi,

I am trying to mimic your results and am getting the following error:

Any idea what could be causing this error?

Note that I have the latest version of torch (2.0.1), not 1.8 because I couldn't find its build anywhere.

Also, the warning about RobertaModel not initialized from the checkpoint -> what does that mean?

Thank you for your help!

"MELD_models/roberta-large/pretrained/no_freeze/emotion/1.0/model.bin" . It doesn't have mobel.bin file.

sorry to disturb you. I have trouble about "CoMPM: Context Modeling with Speaker’s Pre-trained Memory Tracking for Emotion Recognition in Conversation".

when I run the command "python train.py".

it is reported the issue as follows: "MELD_models/roberta-large/pretrained/no_freeze/emotion/1.0/model.bin" . It doesn't have mobel.bin file.

could you please help me to solve it? I want to reproduce your work based on your code.
many thanks
best wishes

RoBERTa large max input token size issue

Question

What if all previous utterances is over than 512 number of tokens?

I know the max input token size of RoBERTa large is 512. [link]

And in your paper, I could find this

context embedding module(CoM) reflects all previous utterances as context.
...
We use an Transformer encoder as a context model (such as RoBERTa).

If you met that kind of problem, then did you use sliding window or something

Thank you

Google Drive link not found

I clicked the link, but the message is like below.

"404. 오류가 발생했습니다. 요청하신 URL을 서버에서 찾을 수 없습니다. 다른 원인은 확인할 수 없습니다."

Could you fix this problem?

Why don't put batch_speaker_token on cuda?

I am reading your code. And I found that, batch_speaker_tokens are not put on cuda. (The code is as below: )

    batch_input_tokens, batch_labels, batch_speaker_tokens = data
    batch_input_tokens, batch_labels = batch_input_tokens.cuda(), batch_labels.cuda()

However, I found the speaker_token is a list with only one Tensor. I am quite confused that, shall we remove the list, just use Tensor as batch_speaker_tokens? In this way, the speaker_tokens can be speed up by GPU.

By the way, I am wondering why you put a list outside the Tensor in batch_speaker_tokens. I think there might be a reason for doing so.

Many thanks.

UnboundLocalError: local variable 'DATA_loader' referenced before assignment

when i run the command "python train.py --initial {pretrained} --cls {emotion} --dataset {MELD}" .

it is reported the error as follows: "UnboundLocalError: local variable 'DATA_loader' referenced before assignment " .

how to fix it ？
many thanks