Error while reproducing the results. about c2-action-detection

[antoninofurnari]

Hello,

The line returning the error is this one:

https://github.com/epic-kitchens/epic-kitchens-slowfast/blob/master/slowfast/datasets/epickitchens_record.py#L34

return int(round(timestamp_to_sec(self._series['stop_timestamp']) * self.fps))

Apparently, you get the error when trying to access self._series['stop_timestamp'], where _series has been loaded from the specified pickle file (see https://github.com/epic-kitchens/epic-kitchens-slowfast/blob/158156a74bd088eee6fe15cfc1d2f41fc85d1509/slowfast/datasets/epickitchens.py#L70).

Can you try to load the pickle file you specified ('/home/user/puri/va/C2-Action-Detection/BMNProposalGenerator/output/ek100/result_proposal-test.pkl') and see if it actually contains the stop_timestamp column? You could do it as follows:

import pandas as pd
data = pd.read_pickle('/home/user/puri/va/C2-Action-Detection/BMNProposalGenerator/output/ek100/result_proposal-test.pkl')
print(data.head())

You should see something like this:

participant_id video_id  start_seconds  stop_seconds start_timestamp stop_timestamp  start_frame  stop_frame     score
narration_id
P07_104_0               P07  P07_104      80.967167     81.452000      0:01:20.96     0:01:21.45         4049        4073  0.641398
P07_104_1               P07  P07_104      80.724750     81.209583      0:01:20.72     0:01:21.20         4037        4061  0.581054
P07_104_2               P07  P07_104      80.967167     81.209583      0:01:20.96     0:01:21.20         4049        4061  0.523063
P07_104_3               P07  P07_104      81.209583     81.694417      0:01:21.20     0:01:21.69         4061        4085  0.517933
P07_104_4               P07  P07_104      81.209583     81.452000      0:01:21.20     0:01:21.45         4061        4073  0.466026

Best,
Antonino

[kajal-puri]

Hi Antonio,

Thanks for the reply. I just did, as u specified and it gave me the following output which doesn't contain the start_timestamp and stop_timestamp section. So I believe I've downloaded the wrong pkl file? Can you point me out the correct one? I believe I've followed the link in the github ReadMe.

participant_id video_id start_seconds stop_seconds
narration_id
P07_104_0 P07 P07_104 80.967167 81.452000
P07_104_1 P07 P07_104 80.724750 81.209583
P07_104_2 P07 P07_104 80.967167 81.209583
P07_104_3 P07 P07_104 81.209583 81.694417
P07_104_4 P07 P07_104 81.209583 81.452000

                       start_frame  stop_frame     score  

narration_id
P07_104_0 4049 4073 0.641398
P07_104_1 4037 4061 0.581054
P07_104_2 4049 4061 0.523063
P07_104_3 4061 4085 0.517933
P07_104_4 4061 4073 0.466026

[antoninofurnari]

You should try with the pickle included in this repo:

https://github.com/epic-kitchens/C2-Action-Detection/blob/master/BMNProposalGenerator/output/ek100/result_proposal-test.pkl

Otherwise, you should be able to generate it yourself by following the instructions at https://github.com/epic-kitchens/C2-Action-Detection/#action-proposals-generator

Hope this solves the issue.

Best,
Antonino

[kajal-puri]

https://github.com/epic-kitchens/C2-Action-Detection/blob/master/BMNProposalGenerator/output/ek100/result_proposal-test.pkl

I have just downloaded the pkl file again (tried on both test and validation) from the above mentioned link and it doesn't include the start_timestamp and stop_timestamp section. Can you download it yourself and check again from this link? I'm trying to generate them myself using the other step that you've mentioned and will let you know if it works!

Thank you for the reply.

[antoninofurnari]

I just realized that the pkl files on the repo where in an outdated format which did not contain the start_timestamp and stop_timestamp columns. I now uploaded the updated files. This should be solved in fafd68d.

Can you try with the updated files?

Thanks,
Antonino

[kajal-puri]

Thanks for updating the pkl files. They work fine now, I just checked.

After updating these, I'm getting the following error. Do you have any idea why this error could be happening?

[INFO: epickitchens.py: 43]: Constructing EPIC-KITCHENS test...
[INFO: epickitchens.py: 77]: Constructing epickitchens dataloader (size: 1454740) from ['/home/user/puri/va/C2-Action-Detection/BMNProposalGenerator/output/ek100/result_proposal-test.pkl']
[INFO: run_net.py: 202]: Testing model for 1454740 iterations
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Traceback (most recent call last):
File "run_net.py", line 387, in
main()
File "run_net.py", line 382, in main
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
Reading failed. Will retry.
test(cfg=cfg)
File "run_net.py", line 232, in test
Reading failed. Will retry.
preds, labels, metadata = perform_test(test_loader, model, test_meter, cfg)
File "run_net.py", line 49, in perform_test
for cur_iter, (inputs, labels, video_idx, meta) in enumerate(test_loader):
File "/home/user/puri/miniconda3/envs/c2-action-detection-bmn/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/user/puri/miniconda3/envs/c2-action-detection-bmn/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/home/user/puri/miniconda3/envs/c2-action-detection-bmn/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/home/user/puri/miniconda3/envs/c2-action-detection-bmn/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
raise self.exc_type(msg)
Exception: Caught Exception in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/user/puri/miniconda3/envs/c2-action-detection-bmn/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/user/puri/miniconda3/envs/c2-action-detection-bmn/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/user/puri/miniconda3/envs/c2-action-detection-bmn/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/user/puri/va/C2-Action-Detection/SlowFastProposalClassifier/slowfast/datasets/epickitchens.py", line 129, in getitem
frames = pack_frames_to_video_clip(self.cfg, self._video_records[index], temporal_sample_index)
File "/home/user/puri/va/C2-Action-Detection/SlowFastProposalClassifier/slowfast/datasets/frame_loader.py", line 44, in pack_frames_to_video_clip
frames = utils.retry_load_images(img_paths)
File "/home/user/puri/va/C2-Action-Detection/SlowFastProposalClassifier/slowfast/datasets/utils.py", line 36, in retry_load_images
raise Exception("Failed to load images {}".format(image_paths))
Exception: Failed to load images ['/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004049.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004050.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004052.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004054.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004055.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004057.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004059.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004060.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004062.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004064.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004065.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004067.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004069.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004070.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg']

[antoninofurnari]

Hello,

That error means that there was an issue reading at least one of the frames in the list. I'd say a good starting point to debug is to check whether those images actually exist. I guess you can check it by doing ls filename.jpg in a bash shell for all files in the list:

['/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004049.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004050.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004052.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004054.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004055.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004057.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004059.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004060.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004062.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004064.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004065.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004067.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004069.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004070.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg', '/home/user/puri/EPIC-KITCHENS//P07/rgb_frames/P07_104/frame_0000004071.jpg']

Since this code is based on the epic-kitchens-slowfast repo, if you have not already done it, it could be a good idea to follow the instructions at https://github.com/epic-kitchens/epic-kitchens-slowfast and check if you can correctly validate the model on the validation set. If you can do that, then there must be an issue with this repo and we will try to solve it together.

Best,
Antonino

[kajal-puri]

Thanks for the reply!

I just checked and you're right these files do not exist. I am using this link mentioned on the SLOWFAST repo to download the datasets and as mentioned here there are two directories created by the name of P01 and P27 whereas the missing frames addressed are in P07 which doesn't exist in the dataset that I have downloaded. Do you know if the dataset name has been changed from P07 to P01 or something that I'm missing here?

[antoninofurnari]

Hello,

After following the instructions of the epic-kitchens-download-scripts repo you should end up with many more than two directories. Each of the directories contains frames of video of one of the subjects, so you should end up with many more folder (one per subject).

May it be that you only launched the python epic_downloader.py --errata command? This will download only files for which we provided some updates a few months ago. You should download all frames with, e.g., python epic_downloader.py --rgb-frames.

Can you try that and see if it solves the issue?

[kajal-puri]

Apologies for the late response.

Is it necessary to download the whole dataset (more than 500 GB) in order to perform the testing as well? Generally, there is a "test" directory in which all the test/demo data is stored so that it is easy to reproduce a few results. As I'm using the pre-trained model, is there any such option available where I can only download the required data for testing and not the whole dataset (as I have already downloaded the dataset/video features for each video)? Let me know your thoughts. Thank you!

[antoninofurnari]

Hello,

It should not be necessary to download the whole dataset if you only intend to perform testing. You should refer to the epic-kitchens-download-scripts repo to see if there an option to only download test videos. In case you can't find one, you could open an issue there to receive guidance.

Hope this helps.