jonasrohw / mai-project

relevant-evidence-detection

Original paper: https://doi.org/10.48550/arXiv.2311.09939 Orignal Implementation: https://github.com/stevejpapad/relevant-evidence-detection

Reproduce

• Clone this repo:

git clone https://github.com/jonasrohw/mai-project
cd mai-project
python src/main.py

• Create a python (>= 3.9) environment (Anaconda is recommended)
• Install all dependencies with: pip install -r requirements.txt.

Datasets

If you want to reproduce the experiments on the paper it is necessary to first download the following datasets and save them in their respective folder:

• VisualNews -> https://github.com/FuxiaoLiu/VisualNews-Repository -> data/VisualNews/
• NewsCLIPings -> https://github.com/g-luo/news_clippings -> data/news_clippings/
• NewsCLIPings evidence -> https://github.com/S-Abdelnabi/OoC-multi-modal-fc -> data/news_clippings/queries_dataset/
• VERITE -> https://github.com/stevejpapad/image-text-verification -> data/VERITE/ (Although we provide all necessary files as well as the external evidence)

Folders:

├── README.md
├── checkpoints_pt
├── create-subset.py
├── data
│   ├── VERITE
│   ├── VisualNews
│   │   └── origin
│   └── news_clippings
│       └── queries_dataset
├── requirements.txt
├── results
├── setup_mnt_data.sh
├── src
│   ├── experiment.py
│   ├── extract_features.py
│   ├── main.py
│   ├── models.py
│   ├── models_dynamic.py
│   ├── prepare_VERITE.py
│   ├── prepare_evidence.py
│   ├── utils.py
│   └── utils_evidence.py
└── verite-healing.py

mai-project's Issues

Modality fusion method: Cross Attention between features and evidences

In this modality fusion method, we try to improve the model by applying cross attention to the features and evidences.
By doing that we hope that we get representations that have encoded more complex releationships between the features and evidences.

The modifications to make in the codebase:

• models.py
  -- implementation of the CrossAttention module and StackedCrossAttention module

• utils.py
  -- prepare_input(): calling the StackedCrossAttention module and applying cross attention to the features/evidences
  -- eval_verite(), train_step(), eval_step(): all of these functions use the prepare_input funtion

• run_experiment.py
  -- included new parameters: num_heads_options=4, dropout_options=0.2, num_layers_options=3,
  weight_decay=1e-3
  -- increased early_stopping from 10 to 20
  -- initialization of the StackedCrossAttention module
  -- init weights of the model with xavier weights
  -- modified optimizer such that the parameters of the StackedCrossAttention module also gets optimized and
  -- included weight_decay -> mitigates overfitting
  -- implemented CyclicLR scheduler that increases the learning rate step by step after each epoch, baseLR = 1e-5 and
  max_lr = 1e-3