scofield7419 / thor-isa

Codes for ACL 2023 paper: Reasoning Implicit Sentiment with Chain-of-Thought Prompting

Hi there     Coding hard

thor-isa's Issues

Question on Reasoning Enhancement Reproducibility

Dear @scofield7419,

According to the original experiment, it was mentioned that highest results were obtained vith Self-Consistency as follows:

Since the implementation represent a pure contribution from the prospect of THoR, I belive that such technique was not sources in the repository.
If so, I would be delighted if you be able to provide a bit more details onto the following questions:

1. Are there any parameters were affected to vary the reasoning paths of Flan-T5 at generation stage?
2. For aspect (a) and opinion (o), how the voting is applicable?

Thank you very much for your time!

Questions on labels alignment to integet values in config

Dear, @scofield7419

I have a question related to alignment in .yaml configuration towards their integer values:

THOR-ISA/config/config.yaml

Line 6 in f6138d5

label_list: ['positive', 'negative', 'neutral']

This is specifically for the the thor and prompt zero-shot and traning cases, since GPT-related part is tend to be not relies on order at all.

In particular, here is how we map processed integer labels.

THOR-ISA/src/loader.py

Line 112 in f6138d5

labels = [self.config.label_list[int(w)] for w in input_labels]

From this code I would assume that we deal with integer values in input that are aligned with the order in config.
After a deeper investigation of the preprocessing code, I could not find any conversions except of reading such values as it is:

THOR-ISA/src/loader.py

Line 161 in f6138d5

label = cur_data['labels'][i]

So that I beleive it is crucial that integer labels supposed to be aligned with their string versions order in config.
If this so, then it means that orginal task should have: 0 as positive, 1 for negative and 2 for neutral.
I believe that is something that is not affected your results and experiments. Would it be possible an explain the history that stays behind the label_list parameter, and am I right that everything described above is expected behavior?

PS: there are other parts where, we assign the default values which is 0 which would refer to positive label in config:

THOR-ISA/src/model.py

Line 36 in f6138d5

output = [label_dict.get(w.replace('<pad>', '').replace('</s>', '').strip(), 0) for w in dec]

Thank you very much for your time!

error in zero_shot

when I try to make the arg "-z" False to carry out main.py,it shows an error named "main.py: error: unrecognized arguments: False"

Fail to run source code

I followed README.md to set up environments. But when I tried to run the code with:

python main.py -f C:\Users\inlab\Desktop\THOR-ISA\config\config.yaml

in

(thor) C:\Users\inlab\Desktop\THOR-ISA>

Error information with Pycharm 2023.1

Running on the laptops data.
Choosing prompt one-step infer mode.
Zero-shot mode for evaluation.
  0%|                                                                                                                                                                                                                                                                                            | 0/1 [00:00<?, ?it/s] 
Traceback (most recent call last):
  File "main.py", line 69, in <module>
    template.forward()
  File "main.py", line 45, in forward
    r = trainer.evaluate_step(self.testLoader, 'test')
  File "C:\Users\inlab\Desktop\THOR-ISA\src\engine.py", line 79, in evaluate_step
    for i, data in tqdm(enumerate(dataiter), total=dataLoader.data_length):
  File "C:\Users\inlab\.conda\envs\thor\lib\site-packages\tqdm\std.py", line 1178, in __iter__
    for obj in iterable:
  File "C:\Users\inlab\.conda\envs\thor\lib\site-packages\torch\utils\data\dataloader.py", line 521, in __next__
    data = self._next_data()
  File "C:\Users\inlab\.conda\envs\thor\lib\site-packages\torch\utils\data\dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\inlab\.conda\envs\thor\lib\site-packages\torch\utils\data\_utils\fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "C:\Users\inlab\Desktop\THOR-ISA\src\loader.py", line 87, in collate_fn
    input_tokens, input_targets, input_labels, implicits = zip(*data)
TypeError: 'int' object is not iterable

my virtual environment named thor

python

Python 3.8.17

pip list in thor

Package            Version
------------------ ------------
attrdict           2.0.1
certifi            2023.5.7
charset-normalizer 3.2.0
colorama           0.4.6
filelock           3.12.2
fsspec             2023.6.0
huggingface-hub    0.16.4
idna               3.4
Jinja2             3.1.2
joblib             1.3.1
MarkupSafe         2.1.3
mpmath             1.3.0
networkx           3.1
numpy              1.24.4
packaging          23.1
pandas             2.0.3
Pillow             10.0.0
pip                23.1.2
python-dateutil    2.8.2
pytz               2023.3
PyYAML             6.0
regex              2023.6.3
requests           2.31.0
safetensors        0.3.1
scikit-learn       1.3.0
scipy              1.10.1
setuptools         67.8.0
six                1.16.0
sympy              1.12
threadpoolctl      3.1.0
tokenizers         0.13.3
torch              1.10.0+cu113
torchaudio         0.10.0+cu113
torchvision        0.11.0+cu113
tqdm               4.65.0
transformers       4.29.0
typing_extensions  4.7.1
tzdata             2023.3
urllib3            2.0.3
wheel              0.38.4

transformers

>>> import transformers
>>> transformers.__version__
'4.29.0'

But in README.md,

apperas just below title, which seems to be contracdictory to requirements.txt, where transformers==4.29.0

my system

• Windows11
• NVIDIA GeForce RTX 3090