artanic30 / maccap

AAAI 2024 Accepted Paper Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training

MacCap

AAAI 2024 Accepted Paper Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training

Setup

First, download and set up the repo:

git clone https://github.com/Artanic30/MacCap
cd MacCap
conda env create -f environment.yml
conda activate MacCap

Data preparation

Download coco_train to data. Download cc3m_train to data.

Training

./train_coco.sh

or

./train_cc3m.sh

Evaluation

Follow the instruction here to evaluate generated captions.

Citation

@article{qiu2024mining,
  title={Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training},
  author={Qiu, Longtian and Ning, Shan and He, Xuming},
  journal={arXiv preprint arXiv:2401.02347},
  year={2024}
}

Acknowledgments

This repository is heavily based on ClipCap, DeCap. For training we used the data of COCO dataset and Conceptual Captions.

Release Schedule

• Initial Code release
• Detail Document
• Data Preparation
• Training and Evaluation Scripts
• Checkpoints

maccap's Issues

where is train_coco.sh

你好，训练代码在哪呀

Request for Checkpoint File

Hi, I am very interested in your code, but I noticed that you haven't released the checkpoint file yet. Would it be possible for you to share the checkpoint file sometime soon? Thanks!

When will the codes will be released?

About the code

Great Job! When will the codes will be released?

How to get the mscoco_test.json?

When I was runing the train_coco.sh, there is a FileNotFoundError:

name 'DeCap' is not defined

thanks for your wonderful work, but when i try to reproduce this work, I facing up this issue in here, is there a mistake?

Reasons for the secondary combination of regional features and attention weights

Great work！I have the following questions, in the last operation to obtain the region feature, the attention weight is the attention weight of the previous layer ? if so, the patch token already obtained by interacting with the attention weight A, is it just to obtain the mutual information between valid patch tokens here?