junyangwang0410 / knight Goto Github PK

SotA text-only image/video method (IJCAI 2023)

Python 96.65% Jupyter Notebook 3.35%

knight's Introduction

README

Paper (Accepted by IJCAI 2023)

1. Installing

$ pip install -r requirements.txt
$ pip install git+https://github.com/openai/CLIP.git

2. Data Preparation

Downloading the images and videos of each dataset from Web.

The data files looks like:

./data/
  ├──./COCO/
  |   ├──./image/					#images of the test split
  |   ├──captions_val2014.json		#annotation of test split
  |   ├──coco_test.txt				#test split of Karpathy
  ├──./Flickr/
  |   ├──./image/					#images in dataset
  |   ├──dataset_flickr30k.json		#annotation
  ├──./MSRVTT/
  |   ├──./video/					#images in dataset
  |   ├──./frames/					#keyframes
  |   ├──train_val_videodatainfo.json	#annotation
  ├──./MSVD/
  |   ├──./video/					#images in dataset
  |   ├──./frames/					#keyframes
  |   ├──caption.txt				#annotation
  |   ├──train_list.txt				#train split
  |   ├──test_list.txt				#train split

After preparing the data, execute the following commands to obtain the data files required to run

python data_prepare_{dataset name}.py

dataset name = {coco, flickr, msrvtt, msvd}

3. Run

Image Captioning

python run_image_captioning.py --dataset {dataset name}

dataset name = {coco, flickr}

Video Captioning

python run_video_captioning.py --dataset {dataset name}

dataset name = {msrvtt, msvd}

The default save path for checkpoints is ./checkpoint/{dataset name}, and the default save path for caption flies is ./output/{dataset name}, where the dataset name = {coco, flickr, msrvtt, msvd}

4. Evaluation

We provide the reference results and the results generated as the paper under the ./output/{dataset_name}/

For example:

python evalution.py 
--ref ./output/COCO/reference_COCO.json
--gts ./output/COCO/result_COCO.json

5. Demo

Getting the checkpoint as above operations and put them in ./checkpoint/COCO/ as:

./checkpoint/
  ├──./COCO/
  |   ├──decoder_coco.pth
  |   ├──map_coco.pth

Then run the demo.ipynb

knight's People

Contributors

Stargazers

Watchers

knight's Issues

python run_image_captioning.py --dataset flickr

Hello, thank you for sharing code.

I have the following error when use Flickr30k on colab.

!python run_image_captioning.py --dataset flickr
FileNotFoundError: [Errno 2] No such file or directory: './feature/Flickr/nibers.npy'

about running run_image_captioning.py --dataset coco

Thanks for amazing work.
when I run
python running run_image_captioning.py --dataset coco
some error occurs. the error code is shown as follow:
Traceback (most recent call last): File "run_image_captioning.py", line 148, in <module> main(args) File "run_image_captioning.py", line 97, in main output = GPT_model(**token, labels = token["input_ids"], prefix = batch_caption_feature) File "/home/boyang/anaconda3/envs/knight/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) TypeError: forward() got an unexpected keyword argument 'prefix'
https://github.com/junyangwang0410/Knight/blob/e03dc2e340abcf418aba711acc300946145a0b08/run_image_captioning.py#LL97C25-L97C25

my conda environment is show as follow:

请教一下数据集中的coco_test.txt是什么内容？我从官网下载的coco2014 image captioning 数据集中没有看到这个。

    json_path = "./data/COCO/captions_val2014.json"
    json_labels = json.load(open(json_path,'r'))
    annotations = json_labels["annotations"]
    images = json_labels["images"]
    images_path = "./data/COCO/image/"

    image_dict = dict()
    for image in images:
        image_dict[image["file_name"]] = image["id"]

    with open("./data/COCO/coco_test.txt") as image_names_data:
        image_names = image_names_data.readlines()

    image_features = []
    for image_info in image_names:
        image_file = image_info.split('\n')[0]
        image_id = image_dict[image_file]
        image_path = images_path + image_file
        ori_image = Image.open(image_path)
        image = preprocess(ori_image).unsqueeze(0).to(device)
        image_feature = clip_model.encode_image(image)
        image_features.append(image_feature)
        
    image_features = torch.cat(image_features)
    torch.save(image_features, "./feature/COCO/image_features.pkl")

因为不懂这个coco_test.txt文件，这段代码没有看明白，如果是读取图片的话，应该只需要拼接file_name与image_folder_name吧。

请教一下generate()方法中prefix参数。

def caption_generation(image_feature, model: GPT2LMHeadModel, tokenizer, device):
	text = "prefix prefix prefix prefix prefix:"
	inputs = tokenizer(text, return_tensors="pt")
	output = model.generate(inputs["input_ids"].to(device), 40, prefix = image_feature, do_sample = False, num_beams=5)[0]
	output = tokenizer.decode(output)
	return output.split(':')[1].split('.')[0].lower()

如上这段代码model.generate()方法中用到了一个prefix参数，我在查阅Huggingface的文档中并没有找到关于prefix参数的解释。

在modeling_gpt2.py文件中，我找到了如下部分代码：

def forward(
        ...
        prefix: Optional[torch.FloatTensor] = None,
    ) -> Union[Tuple, BaseModelOutputWithPastAndCrossAttentions]:
        ...

以及：

...
if inputs_embeds is None:
    inputs_embeds = self.wte(input_ids)
if prefix != None:
    prefix = prefix.expand(inputs_embeds.shape[0], 5, inputs_embeds.shape[2])
    inputs_embeds = torch.cat((prefix, inputs_embeds[:, 5:, :]), dim = 1)
position_embeds = self.wpe(position_ids)
hidden_states = inputs_embeds + position_embeds
...

这段部分的添加应该是作者的修改对吗？期待您的回复。