yalesong / pvse Goto Github PK

I have read your article and I am very interested in your paper. But there are some problems when I repeat your experiment: when I use the coco dataset to evaluate the pretrained model, I find that the results are not consistent with the results you mentioned in the paper. The only difference is that I use Torch==1.2.0, torchvison==0.4.0. Here are my results:

H:***\Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval\pvse-master>python eval.py --data_name coco --num_embeds 2 --img_attention --txt_attention --ckpt ./ckpt/coco_pvse.pth

Loading dataset
loading annotations into memory...
Done (t=0.41s)
creating index...
index created!
Computing results... (eval_on_gpu=CPU)
Images: 5000, Sentences: 25000
Image to text: 0.00, 0.30, 0.60, 773.00 (0.03), 1016.47 (0.04)
Text to image: 0.14, 0.52, 1.00, 500.00 (0.02), 500.83 (0.02)
rsum: 2.56 ar: 0.30 ari: 0.55
Image to text: 0.00, 0.60, 1.40, 781.00 (0.03), 1017.42 (0.04)
Text to image: 0.10, 0.48, 0.96, 501.00 (0.02), 500.82 (0.02)
rsum: 3.54 ar: 0.67 ari: 0.51
Image to text: 0.00, 0.20, 0.90, 776.00 (0.03), 987.25 (0.04)
Text to image: 0.14, 0.46, 0.98, 500.00 (0.02), 500.38 (0.02)
rsum: 2.68 ar: 0.37 ari: 0.53
Image to text: 0.10, 0.30, 0.50, 778.00 (0.03), 989.40 (0.04)
Text to image: 0.06, 0.50, 1.04, 501.00 (0.02), 500.60 (0.02)
rsum: 2.50 ar: 0.30 ari: 0.53
Image to text: 0.10, 0.50, 0.70, 778.00 (0.03), 985.18 (0.04)
Text to image: 0.10, 0.50, 1.00, 500.00 (0.02), 500.52 (0.02)
rsum: 2.90 ar: 0.43 ari: 0.53

Mean metrics from 5-fold evaluation:
rsum: 17.02
Average i2t Recall: 0.41
Image to text: 0.04 0.38 0.82 777.20 (0.03) 999.14 (0.04)
Average t2i Recall: 0.53
Text to image: 0.11 0.49 1.00 500.40 (0.02) 500.63 (0.02)
rsum: 0.58
Average i2t Recall: 0.08
Image to text: 0.04 0.06 0.14 3891.00 (0.16) 4999.44 (0.20)
Average t2i Recall: 0.11
Text to image: 0.02 0.10 0.22 2502.00 (0.10) 2501.31 (0.10)

Thank you for reading and I am looking forward to your reply

The following problems occurred when I used the mrw and tgif data set. How can I solve them?
mrw:I:\Python3.6\python.exe "M:/HJH/Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval/pvse-master/eval.py"
Loading dataset
Computing results... (eval_on_gpu=False)
Traceback (most recent call last):
File "M:/HJH/Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval/pvse-master/eval.py", line 290, in
metrics = evalrank(model, args, split='test')
File "M:/HJH/Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval/pvse-master/eval.py", line 214, in evalrank
img_embs, txt_embs = encode_data(model, data_loader, args.eval_on_gpu)
File "M:/HJH/Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval/pvse-master/eval.py", line 28, in encode_data
for i, data in enumerate(data_loader):
File "I:\Python3.6\lib\site-packages\torch\utils\data\dataloader.py", line 819, in next
return self._process_data(data)
File "I:\Python3.6\lib\site-packages\torch\utils\data\dataloader.py", line 846, in _process_data
data.reraise()
File "I:\Python3.6\lib\site-packages\torch_utils.py", line 369, in reraise
raise self.exc_type(msg)
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "I:\Python3.6\lib\site-packages\torch\utils\data_utils\worker.py", line 178, in worker_loop
data = fetcher.fetch(index)
File "I:\Python3.6\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "I:\Python3.6\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "M:\HJH\Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval\pvse-master\data.py", line 211, in getitem
frames, meta = self.gulp[self.ids[index], start_idx:end_idx:skip]
File "I:\Python3.6\lib\site-packages\gulpio\fileio.py", line 135, in getitem
chunk_id = self.chunk_lookup[id]
KeyError: '3R3CSYCwXTmBW'

Process finished with exit code 1

tgif:I:\Python3.6\python.exe "M:/HJH/Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval/pvse-master/eval.py"
Loading dataset
Computing results... (eval_on_gpu=False)
Traceback (most recent call last):
File "M:/HJH/Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval/pvse-master/eval.py", line 290, in
metrics = evalrank(model, args, split='test')
File "M:/HJH/Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval/pvse-master/eval.py", line 214, in evalrank
img_embs, txt_embs = encode_data(model, data_loader, args.eval_on_gpu)
File "M:/HJH/Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval/pvse-master/eval.py", line 28, in encode_data
for i, data in enumerate(data_loader):
File "I:\Python3.6\lib\site-packages\torch\utils\data\dataloader.py", line 819, in next
return self._process_data(data)
File "I:\Python3.6\lib\site-packages\torch\utils\data\dataloader.py", line 846, in _process_data
data.reraise()
File "I:\Python3.6\lib\site-packages\torch_utils.py", line 369, in reraise
raise self.exc_type(msg)
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "I:\Python3.6\lib\site-packages\torch\utils\data_utils\worker.py", line 178, in worker_loop
data = fetcher.fetch(index)
File "I:\Python3.6\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "I:\Python3.6\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "M:\HJH\Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval\pvse-master\data.py", line 296, in getitem
frames, meta = self.gulp[self.ids[index], start_idx:end_idx:skip]
File "I:\Python3.6\lib\site-packages\gulpio\fileio.py", line 135, in getitem
chunk_id = self.chunk_lookup[id]
KeyError: 'tumblr_n9n04znLpA1qddk8uo1_500'

Process finished with exit code 1

Thanks for your job!
Sorry to bother you again

When I run your eval.py program, no matter how small my batch_size_eval setting is, I find that the graphics memory keeps accumulating, which eventually causes the graphics card to run out of memory and the program fails. My graphics card is GTX1080TI 11G.

Any help would be appreciated. Thank you!

Hello, I successfully loaded the pre-trained model given for mrw dataset using the following command in eval.py script

"checkpoint = torch.load(args.ckpt)
model.load_state_dict(checkpoint, strict=False)"

but the results are not same as given in the article or in github repo.

The results generated by evaluating pre-trained model "mrw_pvse.pth"

Loading dataset
Computing results... (eval_on_gpu=True)
Images: 5000, Sentences: 5000
rsum: 0.48
Average i2t Recall: 0.07
Image to text: 0.02 0.06 0.14 2509.00 (0.50) 2509.74 (0.50)
Average t2i Recall: 0.09
Text to image: 0.02 0.06 0.18 2500.00 (0.50) 2500.98 (0.50)

Am I doing any mistake

Thanks for sharing this great work.
I found that there is a GPU memory leak during the forward pass in eval.py (line 34).
This prevents to complete the evaluation as it crashes at about 7700 samples on a 24GB GPU.
Best regards,
Georges.

When I run python eval.py --data_name mrw --num_embeds 1 --img_attention --txt_attention --max_video_length 4 --ckpt ./ckpt/mrw_pvse_k1.pth. Something wrong occured.

(pvse) zhyue@server-55:~/pvse$ python eval.py --data_name mrw --num_embeds 1 --img_attention --txt_attention --max_video_length 4 --ckpt ./ckpt/mrw_pvse_k1.pth
Loading dataset
Computing results... (eval_on_gpu=False)
Traceback (most recent call last):
  File "eval.py", line 290, in <module>
    metrics = evalrank(model, args, split='test')
  File "eval.py", line 214, in evalrank
    img_embs, txt_embs = encode_data(model, data_loader, args.eval_on_gpu)
  File "eval.py", line 28, in encode_data
    for i, data in enumerate(data_loader):
  File "/home/zhyue/anaconda2/envs/pvse/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/home/zhyue/anaconda2/envs/pvse/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 606, in _process_next_batch
    raise Exception("KeyError:" + batch.exc_msg)
Exception: KeyError:Traceback (most recent call last):
  File "/home/zhyue/anaconda2/envs/pvse/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/zhyue/anaconda2/envs/pvse/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/zhyue/pvse/data.py", line 211, in __getitem__
    frames, meta = self.gulp[self.ids[index], start_idx:end_idx:skip]
  File "/home/zhyue/anaconda2/envs/pvse/lib/python3.7/site-packages/gulpio/fileio.py", line 135, in __getitem__
    chunk_id = self.chunk_lookup[id_]
KeyError: '3R3CSYCwXTmBW'

Any suggestion?

Thanks for your job!
I try to reproduce your job and I get confused in the word embedding.
I use torchtext 0.4.0 and find 70 words missing in GloVe in Coco.
I got very bad results after training 30 epoch with the same parameters.
Though I use torch 1.2.0 and torchvision 0.4.0, I don't think they are the main reasons.
I want to know how you initialize the missing word embeddings or other problems you think I am facing.
Any help would be appreciated. Thank you!

I'm currently porting your code over to run a comparison against it. I came across the following issue in model.py:

Let's modify the code above to preserve the final hidden state of the RNN.

>>> rnn_out, h_n = self.rnn(packed) 

Now, let's inspect. (Note, the batch size may differ in your case, as well as the hidden state size from my implementation):

>>> h_n.shape
torch.Size([2, 32, 256])
>>> padded[0].shape
torch.Size([32, 15, 512])

So, we see h_n, which is the hidden state after consuming the final input, has 2 in its first dimension, since we have a bidirectional GRU. We also observe (in my case), the hidden state size is 256, while the output is size 512 (this is because Pytorch concats the two BiGRU (forward-backward) states in the output).

Let's examine the 5th element in the batch. First, we find its length:

>>> padded[1][5]
tensor(11)

So, we know the final hidden state occurs at index 10 (and the rest are padded). We can confirm here:

>>> torch.equal(padded[0][5, 10, :256], h_n[0, 5, :])
True

Here is where the problem comes:

>>> torch.equal(padded[0][5, 10, 256:], h_n[1, 5, :])
False

In other words, what we find is that the final hidden state for the reverse direction h_n[1, 5, :], does not equal the hidden state at padded[0][5, 10, 256:].

Similarly, we can verify this in your out variable:

>>> out.shape
torch.Size([32, 512])
>>> torch.equal(out[5,:256], h_n[0, 5, :])
True
>>> torch.equal(out[5,256:], h_n[1, 5, :])
False

In other words, the first part of out matches h_n, but the second half does not.

This is because the final hidden state for the reverse direction of the GRU is not at padded[0][5, 10, 256:]. Because it is reversed, it is actually located at index 0 in padded, not at index 10!

>>> torch.equal(padded[0][5, 10, 256:], h_n[1, 5, :])
False
>>> torch.equal(padded[0][5, 0, 256:], h_n[1, 5, :])
True

In other words, I believe your implementation doesn't use the correct hidden state in the reverse direction. If your reverse GRU is processing sequence S_n, ... S_1 and producing hidden states h_1, ... h_n, your implementation only processes the last item of the sequence S_n, and ignores the rest of the sequence (in the reverse direction). The forward direction's output hidden state captures the entire input, but the reverse state hidden state only captures the last item of the input, and none of the prior input states. So, the two hidden states used aren't even capturing the same sequence.

Long story short, I think you want torch.equal(out[5,256:], h_n[1, 5, :]) to be equal, but it isn't currently.

I believe the issue is also present in VideoEncoder, here:

Again, it seems you are using the last index, for both directions of the GRU. This is correct for the forward case, but not backward.

I try to reproduce your model on coco using the command:

python3 train.py --data_name coco --cnn_type resnet152 --wemb_type glove --margin 0.1 --max_violation --num_embeds 2 --img_attention --txt_attention --mmd_weight 0.01 --div_weight 0.1 --batch_size 256

with the pytorch 1.1.0, torchvision 0.3.0 on single RTX2080Ti.

But the loss stays at 0.2001 since epoch 4, and the final performance is bad.
The evaluation result of your provided checkpoint is OK, so I don't think it's the problem with the environment.
Is there any suggestion on this problem?
Any help would be appreciated. Thank you!

First up, thank you so much for the wonderful code. I still have one issue which I am not able to understand from the code. In the code we are dealing with MIL loss but I am not sure how it is implemented using the MaxPool idea. This may be partly due to my lack of understanding about MIL itself. If there is any resource which you think can help me here, I shall be really grateful.

Hi,
In MIL loss part, you use cosin distance d(a, b)=(a dot b)/(norm(a),norm(b)) , this experssion means if a is similar with b , the d(a,b) is become large. Do you want to express this mean in MIL loss part?
If I understand wrong, please correct me. I am confused with this part.

how to understand residual eliminate redundant information. Hope to get your recovery

thank you for your awesome work! when train the MRW dataset , I encounter the following problem:
File "train.py", line 236, in
main()
File "train.py", line 216, in main
loss = train(epoch, trn_loader, model, criterion, optimizer, args)
File "train.py", line 68, in train
for itr, data in enumerate(data_loader):
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 560, in next
batch = self.collate_fn([self.dataset[i] for i in indices])
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 560, in
batch = self.collate_fn([self.dataset[i] for i in indices])
File "/root/CV/pvse/data.py", line 230, in getitem
video = self.transform(frames)
File "/root/CV/pvse/video_transforms.py", line 58, in call
img = t(img)
File "/root/CV/pvse/video_transforms.py", line 588, in call
i, j, h, w = self.get_params(img_list[0], self.scale, self.ratio)
IndexError: list index out of range
How to fix it?thank you for your attention

I try to reproduce your job on coco dataset by training from scratch based on the suggested settings.
Namespace(batch_size=128, batch_size_eval=32, ckpt='', cnn_type='resnet152', crop_size=224, data_name='coco', data_path='***', debug=False, div_weight=0.1, dropout=0.0, embed_size=1024, eval_on_gpu=False, grad_clip=2.0, img_attention=True, img_finetune=True, legacy=True, log_file='***', log_step=10, logger_name='***', lr=2e-04, margin=0.1, max_video_length=1, max_violation=True, mmd_weight=0.01, num_embeds=2, num_epochs=10, order=False, txt_attention=True, txt_finetune=True, val_metric='rsum', vocab_path='***', weight_decay=0.0, wemb_type='glove', word_dim=300, workers=16)

However, I find that triplet ranking loss is converged to the margin and the final performance is bad.
How can you fix it? Any help would be appreciated. Thank you!

I am able to replicate the results but found something strange. I made a small tweak and also plotted the loss curves for the training and validation set. While doing so, I get a "strange" tuning-fork shaped loss. Although, the recall sum looks perfectly fine. Can you please tell me if this is the expected behavior for the same?

Your help shall be highly appreciated

When I run python python download_gulp_mrw.py ,some data url is invalid.

jd4lXngPg4Z0c mp4 url is empty
2H8IBNL2Ae9kQ mp4 url is empty
XKyqwYgBLxha8 mp4 url is empty
http://media1.giphy.com/media/c84Hf3KIsQ1uo/giphy is no longer available!
http://media0.giphy.com/media/Y5lzSqen6Y8nK/giphy is no longer available!

I observed that the default epoch is 30, so I am afraid that the model may overfit when training. So I would like to ask if this model has a mechanism to prevent overfitting?

Hello, First of all thanks alot for sharing such a great work. I used your code to train the model on MSR_VTT dataset, and I did that successfully, thanks for clear code and instructions. Now, I have "model_best.pth.tar" file. I also downloaded your model mrw_pvse.pth from github repo. Running eval.py is giving an error on loading model from a given path.

Even tried to load your model "mrw_pvse.pth" but the same error

" RuntimeError: Error(s) in loading state_dict for PVSE:
Missing key(s) in state_dict: "img_enc.cnn.conv1.weight", "img_enc.cnn.bn1.weight", "img_enc.cnn.bn1.bias", "img_enc.cnn.bn1.running_mean", "img_enc.cnn.bn1.running_var", "img_enc.cnn.layer1.0.conv1.weight", "img_enc.cnn.layer1.0.bn1.weight", "img_enc.cnn.layer1.0.bn1.bias", "img_enc.cnn.layer1.0.bn1.running_mean", "img_enc.cnn.layer1.0.bn1.running_var", "img_enc.cnn.layer1.0.conv2.weight", "img_enc.cnn.layer1.0.bn2.weight", "img_enc.cnn.layer1.0.bn2.bias", "img_enc.cnn.layer1.0.bn2.running_mean", "img_enc.cnn.layer1.0.bn2.running_var", "img_enc.cnn.layer1.0.conv3.weight", "img_enc.cnn.layer1.0.bn3.weight", "img_enc.cnn.layer1.0.bn3.bias", "img_enc.cnn.layer1.0.bn3.running_mean", "img_enc.cnn.layer1.0.bn3.running_var", "img_enc.cnn.layer1.0.downsample.0.weight", "img_enc.cnn.layer1.0.downsample.1.weight", "img_enc.cnn.layer1.0.downsample.1.bias", "img_enc.cnn.layer1.0.downsample.1.running_mean", "img_enc.cnn.layer1.0.downsample.1.running_var", "img_enc.cnn.layer1.1.conv1.weight", "img_enc.cnn.layer1.1.bn1.weight", "img_enc.cnn.layer1.1.bn1.bias", "img_enc.cnn.layer1.1.bn1.running_mean", "img_enc.cnn.layer1.1.bn1.running_var", "img_enc.cnn.layer1.1.conv2.weight", "img_enc.cnn.layer1.1.bn2.weight", "img_enc.cnn.layer1.1.bn2.bias", "img_enc.cnn.layer1.1.bn2.running_mean", "img_enc.cnn.layer1.1 .... so on"

Kindly guide me with this issue