jjboy / bmn-boundary-matching-network Goto Github PK

The feature length of one frame about RGB and flow must be 400?

In Base Module and Temporal Evaluation Module,all of the conv1d layer group=4,but I haven't say the param in paper.

In default BMN, the temporal scale is 100.
if I want to generate 200*200 proposal map by feature with 200-length, need I change some hyperparameters ?

For instance, need I change the positive threshold of cls_pem_loss or three thresholds of reg_pem_loss ?
Is there anything else that needs to be modified

i download the BMN_best.pth.tar from the BaiduYun(pan.baidu.com) link u provided, for 2 times, both cannot be decompressed.

>tar -xvf BMN_best.pth.tar
tar: Error opening archive: Unrecognized archive format

Could u pls. help to check if the compressed file corrupted?

Hi,

Dowloaded your features, trained and evaluated you model.
I get the following results

train subset video numbers: 9649
validation subset video numbers: 4728
BMN training loss(epoch 0): tem_loss: 1.152, pem_loss: 0.267, total_loss: 1.419
BMN testing loss(epoch 0): tem_loss: 1.184, pem_loss: 0.261, total_loss: 1.445
BMN training loss(epoch 1): tem_loss: 1.095, pem_loss: 0.235, total_loss: 1.329
BMN testing loss(epoch 1): tem_loss: 1.115, pem_loss: 0.231, total_loss: 1.346
BMN training loss(epoch 2): tem_loss: 1.070, pem_loss: 0.223, total_loss: 1.293
BMN testing loss(epoch 2): tem_loss: 1.098, pem_loss: 0.249, total_loss: 1.347
BMN training loss(epoch 3): tem_loss: 1.056, pem_loss: 0.218, total_loss: 1.274
BMN testing loss(epoch 3): tem_loss: 1.103, pem_loss: 0.222, total_loss: 1.325
BMN training loss(epoch 4): tem_loss: 1.042, pem_loss: 0.213, total_loss: 1.255
BMN testing loss(epoch 4): tem_loss: 1.098, pem_loss: 0.225, total_loss: 1.323
BMN training loss(epoch 5): tem_loss: 1.028, pem_loss: 0.210, total_loss: 1.238
BMN testing loss(epoch 5): tem_loss: 1.116, pem_loss: 0.260, total_loss: 1.377
BMN training loss(epoch 6): tem_loss: 1.016, pem_loss: 0.212, total_loss: 1.229
BMN testing loss(epoch 6): tem_loss: 1.101, pem_loss: 0.223, total_loss: 1.324
BMN training loss(epoch 7): tem_loss: 0.999, pem_loss: 0.205, total_loss: 1.204
BMN testing loss(epoch 7): tem_loss: 1.150, pem_loss: 0.222, total_loss: 1.372
BMN training loss(epoch 8): tem_loss: 0.991, pem_loss: 0.204, total_loss: 1.195
BMN testing loss(epoch 8): tem_loss: 1.138, pem_loss: 0.221, total_loss: 1.359
BMN training loss(epoch 9): tem_loss: 0.926, pem_loss: 0.191, total_loss: 1.117
BMN testing loss(epoch 9): tem_loss: 1.163, pem_loss: 0.216, total_loss: 1.379
BMN training loss(epoch 10): tem_loss: 0.901, pem_loss: 0.188, total_loss: 1.089
BMN testing loss(epoch 10): tem_loss: 1.176, pem_loss: 0.215, total_loss: 1.391
BMN training loss(epoch 11): tem_loss: 0.886, pem_loss: 0.187, total_loss: 1.073
BMN testing loss(epoch 11): tem_loss: 1.218, pem_loss: 0.216, total_loss: 1.434
BMN training loss(epoch 12): tem_loss: 0.873, pem_loss: 0.185, total_loss: 1.059
BMN testing loss(epoch 12): tem_loss: 1.237, pem_loss: 0.216, total_loss: 1.453
BMN training loss(epoch 13): tem_loss: 0.863, pem_loss: 0.185, total_loss: 1.048
BMN testing loss(epoch 13): tem_loss: 1.265, pem_loss: 0.217, total_loss: 1.482
BMN training loss(epoch 14): tem_loss: 0.853, pem_loss: 0.183, total_loss: 1.037
BMN testing loss(epoch 14): tem_loss: 1.257, pem_loss: 0.217, total_loss: 1.474

[INIT] Loaded annotations from validation subset.
Number of ground truth instances: 7293
Number of proposals: 472632
Fixed threshold for tiou score: [0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95]
[RESULTS] Performance on ActivityNet proposal task.
Area Under the AR vs AN curve: 63.50907719731249%

Adding an axes using the same arguments as a previous axes currently reuses the earlier instance. In a future version, a new instance will
always be created and returned. Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
"Adding an axes using the same arguments as a previous axes "
AR@1 is 0.31546688605512135
AR@5 is 0.43129027834910183
AR@10 is 0.5023584258878377
AR@100 is 0.7298642533936651

In your paper the Area Under the AR vs AN curve is 67.10 and I'm getting 63.5
and your AR@100 is 75.01 and I'm getting 72.9.

I didn't change your configuration.
Any idea why I'm getting lower results.

Thanks
Ophir

Hi, I am a graduate student who is interseted in TAD and hope more communication.
First, thank you for the implementation of BMN. I find BMN model's structure is not the same as introduced in the paper, so I checked the source code by Tianwei Lin. I think the output dimension of x_1d_p and input dimension of x_3d_p may be changed to 128(hidden_dim_2d) as follows:
# Proposal Evaluation Module self.x_1d_p = nn.Sequential( nn.Conv1d(self.hidden_dim_1d, self.hidden_dim_2d, kernel_size=3, padding=1), nn.ReLU(inplace=True) ) self.x_3d_p = nn.Sequential( nn.Conv3d(self.hidden_dim_2d, self.hidden_dim_3d, kernel_size=(self.num_sample, 1, 1),stride=(self.num_sample, 1, 1)), nn.ReLU(inplace=True) )
Lin's source code in this link:
https://github.com/PaddlePaddle/models/blob/release/1.8/PaddleCV/video/models/bmn/bmn_net.py#L141-L149

Best wishes!

Thanks for your work to implement BMN on PyTorch!

May I ask, what is the model performance that you get?

On my side, the AUC is 64.5% and AR@100 is 73.5. Please find the details below.

I did not change any parameters before running the code. Minor typos were fixed.

+ python main.py --module Evaluation                                                                                                                                        
[INIT] Loaded annotations from validation subset.                                                                                                     
        Number of ground truth instances: 7293                                                                                                                              
        Number of proposals: 472628                                                                                                                   
        Fixed threshold for tiou score: [0.5  0.55 0.6  0.65 0.7  0.75 0.8  0.85 0.9  0.95]                                                                                 
[RESULTS] Performance on ActivityNet proposal task.                                                                                                   
        Area Under the AR vs AN curve: 64.50481283422461%                                                                                                                   
/home/xum/anaconda3/envs/GTAL/lib/python3.7/site-packages/matplotlib/figure.py:98: MatplotlibDeprecationWarning:                                      
Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version,                                                     a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior                               ensured, by passing a unique label to each axes instance.                                                                                                                   
  "Adding an axes using the same arguments as a previous axes "                                                                                       
AR@1 is          0.30287947346770877                                                                                                                                        
AR@5 is          0.43831070889894425                                                                                                                  
AR@10 is         0.5157136980666392                                                                                                                                         
AR@100 is        0.7352392705333882

Maybe Chinese characters need to be handled...

$ sh bmn.sh
  File "main.py", line 136
SyntaxError: Non-ASCII character '\xe9' in file main.py on line 136, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
  File "main.py", line 136
SyntaxError: Non-ASCII character '\xe9' in file main.py on line 136, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

How are we supposed extract the input of mode, that is found in the csv file.

Hi,I use the proposal result of your code, and try to reproduce the detection result follow the original paper. But the mAp result has a gap between original paper. Did you have this problem, and how did you solve that. Thanks!

Dear authors,
Thank you for sharing the code of your recent paper BMN. I was wondering if you could also share the code you used for activity detection on ActivityNet (Results from Table 6 of your paper).
Any guidelines to classify your proposals afterwards, and reproduce the results of your paper?
Thank you for you support,
Best,

Is each video sampled at different intervals
How does the author turn unequal video into equal length? rescaled the feature length of all videos to same length 100,

Hi, thanks for the great work.

For my project, I would like to visualize the the probability actionness score for input video. How can I visualize that sequence? Any help will be appreciated.

Thank you!

谢谢你的工作，但是在你最新的代码中，opt文件中的"video_info_backup.csv"文件和"anet_anno_action_backup.json"文件好像没有提供，使用的还是之前的label文件？
另外一个问题是：在训练过程中，你使用了best_loss变量，按照我的理解，这应该是一个局部变量，每一个epoch都会初始化为一个极大的值，而每一次保存都是会保存当前epoch的参数？请你确认一下。

Hi,
Very appreciated with your great work!
I'm just start to learn the paper and your code. It seems that there is no classification for any proposals snips of the video for this network. I am not quite sure am I right?

Thanks in advance!

Thank you for your code. I find the training process very slow.

i have a little confused about the get of the weight in the BM layer in paper,could you please tell me how does it work.

Thanks a lot for this code!
I simply run the script to train BMN model. However I got PEM loss=NaN , and I find that label confidence is all 0. What 's wrong there?

thks for your work. but I found that sometimes the batchsize is 1, pem = loss occurs. why?

Hi, thanks for the code. I'd like to try this on my own datasets, but when I was extracting features by using TSN, I can only get RGB.jpg and Flow_x_y.jpg. How can I get csv features which is used in your code? (like csv_mean_100/v_oOnKQgQZOZ0.csv) Or can I use jpg to do training and testing?

hello sir，thank you very much for your code, but I want to know if it is possible to train the model without gpu

Hello, May I get your code about AUC-calculating? I didn't find it in class ANET..(I just find some about AR@AN).
Thank you very much!

When I tried to train the model by using nn.DataParallel in pytorch, Runtime error occurred:

RuntimeError: arguments are located on different GPUs

However , I can train BSN by simply parallel model on multiple GPUS.

Do u have any idea about it?

this repo run on ActivityNet with feature shape 400100,RGB200100 concat flow 200100 ,do you know how to make feature temporal dimension to 100?when i extract feature on Thumos 14 ,I get feature 4096L,L is length of video frames/16. should i do a meanpooling on temporal dimension to fix value???

Thanks for the pytorch codes. I've downloaded the dataset and try to train the model. Could you please tell me your training parameters in the opts.py file? Or you're using all the default parameters?
Thanks.

您好，我正在尝试复现THUSMO14的实验。并且根据您在issue上的建议，我参考了G-TAD，但是复现的效果跟论文上的结果有点差距。在issue中您说您能复现论文中的效果，不知道可否请您分享一下在THUMOS14上的实验代码呢？非常感谢！ @JJBOY

Hi,

I was running the BMN code and saw for all the proposals , the "cls_score" and "reg_score" field which is coming from the Proposal Evaluation Module is coming as 0.5 as shown below. So is it not learning ?

在网络模型的输入中没有发现类别label 信息的载入？太奇怪了，这样自己做一个数据集和修改都不知道从何处 下手。 现在我能把数据集做成 json 文件格式的样子，能够包含起始位置，类型信息等。可以指没看见label类别信息的映射~帮忙指点一下，按照我的理解。比如activitynet 有个200类别，最后肯定有一个分类模块。并且输入输出结果映射指导类别上

Since the training used fixed length of the feature's temporal dimension (~100), is that can be used to unfixed the length of the temporal dimension? and how? thanks a lot.

Hello, thanks for the great work. I want to reproduce the detection results. Could you provide the code for calculate mAP at different tIoU thresholds and the average mAP on Anet dataset, please?

I read all of the related papers such as TSN & BSN, but I don't understand what is the "featureFrame" in the data_process.py file. what is the difference between this and video frame? why use "16*len(data)" to calculate featureFrame?

Did you added BMN into into action temporal detection? And what is the result? I just got about 0.24mAP and I don't know is that correct or not. Thanks a lot.

There could be some difference on preprocessing between THUMOS14 and ActivityNet. Anyone reproduce the results on THUMOS14??

Thanks very much for your great work!

I want to try your work. I installed pytorch0.4.1 and python 3.6. When I run
"python main.py --mode train",
got the following error:

++++++++++++++++++++++++
Traceback (most recent call last):
File "main.py", line 204, in
main(opt)
File "main.py", line 178, in main
BMN_Train(opt)
File "main.py", line 87, in BMN_Train
model = torch.nn.DataParallel(model, device_ids=[0, 1]).cuda()
File "C:\Anaconda\envs\py0.4.1\lib\site-packages\torch\nn\parallel\data_parallel.py", line 111, in init
_check_balance(self.device_ids)
File "C:\Anaconda\envs\py0.4.1\lib\site-packages\torch\nn\parallel\data_parallel.py", line 17, in check_balance
dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
File "C:\Anaconda\envs\py0.4.1\lib\site-packages\torch\nn\parallel\data_parallel.py", line 17, in
dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
File "C:\Anaconda\envs\py0.4.1\lib\site-packages\torch\cuda_init.py", line 292, in get_device_properties
raise AssertionError("Invalid device id")
AssertionError: Invalid device id
+++++++++++++++++++++++++++++

I wonder if it is because the cuda version I installed is different from yours. Please advise the cuda version you used.

Thanks

python main.py --module BMN --mode train

train subset video numbers: 9649
validation subset video numbers: 4728

then get stuck

Hi, thanks for your brilliant work!

I've noticed that your program needs the file "video_info_new.csv" in the "./data/activitynet_annotations/" folder to run the evaluation. Could you please tell me how to make this csv file?

Thanks again!

Hey @JJBOY, thanks for your implementation and I found it extremely useful. I now have a problem during training, the PEM loss is decreasing normally, but TEM loss is not really decreasing, even it's increasing. Do you happen to have experience on how this may happen?

Thanks!

Hi！ Thanks for your great work!
I'd like to combine your algorithm with temporal action classification,but I have no idea to do it.Would you please give me some advice?

Thank you for your share.

When I was reading your code I found that the number of out_channel in layer x_1d_p (below) is different from that described in paper and official repo. So I am figuring out why? Would it help improving performance?

1. What is the meaning of featureframe,is it randomly extract from video then extract feature and reshape to (400,100)?
2. i have generated featurefrom video,and i extract per frame from 200ms,the length of feature is unfixed because of unfixed video,how can i make its length to be fixed ?
3. my dataset is hoi video,so the orginal label is (subject,object,relation),but in this project ,the label is action ,how can i present it?
4. the means of temeporal_scale?

it says result_dict not defined in post_processing.py,NameError: name 'result_dict' is not defined
how to solve it

Thank you for the great work!

I tried your training program, and printed the loss after each batch. The loss seems correct in the beginning of the training, then "nan" values appear as shown in the following. Wonder if you guessed the possible reasons.
...
BMN training loss(epoch 0): tem_loss: 1.184, pem class_loss: 0.458, pem reg_loss: 0.024, total_loss: 1.883
BMN training loss(epoch 0): tem_loss: 1.184, pem class_loss: 0.458, pem reg_loss: 0.024, total_loss: 1.884
BMN training loss(epoch 0): tem_loss: 1.184, pem class_loss: nan, pem reg_loss: 0.024, total_loss: nan
BMN training loss(epoch 0): tem_loss: nan, pem class_loss: nan, pem reg_loss: 0.024, total_loss: nan

Thanks!

Good afternoon!
Thank you very much for your work!
It's great that the model is not only on paddlepaddle.

Could you please share model training speed? I'm using a non-standard environment and would like to compare to the "original".

Hi,

I got this problem of data/activitynet_feature_cuhk/csv_mean_100/v_nt4Ag91306U.csv does not exist. It seems that some of the files are missing.Can you please tell me how to solve this problem?

Hi,

Could I share your well-trained models on THUMOS 2014?

By the way, why BMN_best.pth.tar model can not be decompressed? Seems this tar file is broken.

Best Regards,
Jun Li

Thanks for sharing your code. I'd like to change the number of temporal proposals of the model, but I couldn't find this parameter. Would you please tell me the specific location?