ovshake / cobra

Code for COBRA: Contrastive Bi-Modal Representation Algorithm (https://arxiv.org/abs/2005.03687)

COBRA: Contrastive Bi-Modal Representation Algorithm

This repository contains the code for the paper COBRA: Contrastive Bi-Modal Representation Algorithm (ArXiv) by Vishaal Udandarao, Abhishek Maiti, Deepak Srivatsav, Suryatej Reddy, Yifang Yin and Rajiv Ratn Shah.

Methodology

We present a novel framework COBRA that aims to train two modalities (image and text) in a joint fashion inspired by the Contrastive Predictive Coding (CPC) and Noise Contrastive Estimation (NCE) paradigms which preserve both inter and intra-class relationships. We empirically show that this framework reducesthe modality gap significantly and generates a robust and task agnostic joint-embedding space. We outperform existing work on four diverse downstream tasks spanning across seven benchmark cross-modal datasets.

A visualisation of the loss function:

Architecture

Datasets

The 7 datasets used to empirically prove our results are:

1. PKU-XMedia
2. MS-COCO
3. NUS-Wide 10k
4. Wikipedia
5. FakeNewsNet
6. MeTooMA
7. CrisisMMD

t-SNEs

Results

Instructions for running

The code has been tested on Python 3.6.8 and PyTorch 1.5.1.

• Install all the dependencies using the following command:

pip install -r requirements.txt

• Create a folder features to save the trained models
• To train COBRA, use the following command:

python main.py

• To switch between NCE contrastive loss and softmax contrastive loss, change the use_nce flag. To change the number of anchor points and number of negative samples, modify the num_anchors and num_negative_samples respectively.

Queries

In case of any queries, please open an issue. We will respond as soon as possible.

cobra's Issues

dataset

Hello, do you have the download link of xmediafeatures.mat?

the trained models

Can you provide the trained models? Thank you!

Test script for COCO

Hi,
Thank you so much for sharing the work! I am wondering if you can release the test script for coco dataset?

Cross-modal retrieval test settings are wrong

The saved test_pres are both generated from the test data of one modality. They are the same!

Ckpts of a trained network

Dear Abhishek Maiti,

Thank you for sharing your code and paper. I am very much interested in your work and would like to test COBRA for a cross-modal retrieval problem. So, can you kindly provide checkpoints for your model?

Sincerely,
Magauiya