hyseob / pymdnet

MDNet PyTorch implementation

py-MDNet

by Hyeonseob Nam and Bohyung Han at POSTECH

Update (April, 2019)

• Migration to python 3.6 & pyTorch 1.0
• Efficiency improvement (~5fps)
• ImagNet-VID pretraining
• Code refactoring

Introduction

PyTorch implementation of MDNet, which runs at ~5fps with a single CPU core and a single GPU (GTX 1080 Ti).

[Project] [Paper] [Matlab code]

If you're using this code for your research, please cite:

@InProceedings{nam2016mdnet,
author = {Nam, Hyeonseob and Han, Bohyung},
title = {Learning Multi-Domain Convolutional Neural Networks for Visual Tracking},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2016}
}

Results on OTB

• Raw results of MDNet pretrained on VOT-OTB (VOT13,14,15 excluding OTB): Google drive link
• Raw results of MDNet pretrained on Imagenet-VID: Google drive link

Prerequisites

• python 3.6+
• opencv 3.0+
• PyTorch 1.0+ and its dependencies
• for GPU support: a GPU with ~3G memory

Usage

Tracking

 python tracking/run_tracker.py -s DragonBaby [-d (display fig)] [-f (save fig)]

• You can provide a sequence configuration in two ways (see tracking/gen_config.py):
  • python tracking/run_tracker.py -s [seq name]
  • python tracking/run_tracker.py -j [json path]

Pretraining

• Download VGG-M (matconvnet model) and save as "models/imagenet-vgg-m.mat"
• Pretraining on VOT-OTB
  • Download VOT datasets into "datasets/VOT/vot201x"

   python pretrain/prepro_vot.py
   python pretrain/train_mdnet.py -d vot

• Pretraining on ImageNet-VID
  • Download ImageNet-VID dataset into "datasets/ILSVRC"

   python pretrain/prepro_imagenet.py
   python pretrain/train_mdnet.py -d imagenet

pymdnet's Issues

Regarding the mode of the tracking.

Is it not for multiple object tracking? when I run the code it is only tracking single person.

Utilize the network to generate proposals

Hi,

As far as I understand, the code requires the ground truth for the first bounding box. While one can use some method to generate 1st proposals, the network has learned what a, say, ball looks like, how can I utilize that to generate some initial proposals? Moreover, there are several classes that the network can distinguish, a way to tell the network what to look for can be interesting. Let's discuss more.

Best regards,

The precision

@HyeonseobNam , hello, HyeonseobNam, thanks for your python implementation of the MDNet, this is really a wonderful wok. But i still have a question: do you evaluate the model which is provided by you on the OTB-100 dataset? I test it and find that the precision is far lower than the matlab version. I appreciate that if you can test it on the OTB-100 dataset. Thanks very much !

What should be the approach if the object enters frame a bit late and disappears sometimes?

In the groundtruth the frames without object is empty lines. When the objects joins the video a bit late the program starts to draw some rectangles. So, it causes the IOU value to be incorrect. What should be the GT lines when the object is missing?

precision plots and success plots

Hello, how can I achieve the accuracy you show? Is there any special skills? I can't achieve the accuracy you show here when running your code on my computer.

out of memory

about the 44 frame processed ... run out of memory....running
python run_tracker.py -s DragonBaby -d
Any idea why this is happening?

    return torch.cat(inputs, self.dim)
RuntimeError: cuda runtime error (2) : out of memory at /b/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu:66

The problem for FPS

Hello, I ran your code and found that the number of frames in this run is far from the FPS in the paper.

args.seq or args.json

ImageNet-VID dataset

hi,what do you mean the "ImageNet-VID dataset" ? Because I can find the 2015 :http://bvisionweb1.cs.unc.edu/ilsvrc2015/download-videos-3j16.php#vid and the 2017.
Can any guy give me a link of the ImageNet-VID dataset ?thanks a lot.

args.json

elif args.json != '':
    # load config from a json file
    print("run json")
    param = json.load(open(args.json,'r'))
    seq_name = param['seq_name']
    img_list = param['img_list']
    init_bbox = param['init_bbox']
    savefig_dir = param['savefig_dir']
    result_path = param['result_path']
    gt = None

how to run it?

'tuple' object has no attribute 'size'

Hi,

Has anybody find this problem? I went into an error:

File "run_tracker.py", line 164, in run_mdnet
   feat_dim = pos_feats.size(-1)
AttributeError: 'tuple' object has no attribute 'size'

In the code ,the post_feats was defined in line 162:
pos_feats = (model, image, pos_examples), which is a tuple w/o size attribute.

Do you mean pos_feats[-1].size() ? Because pos_feats[-1] is a ndarray.

But in line 63, you called pos_feats.size(0) again. pos_feats(0)'s type is 'model.MDNet' which doesn't have a size attribute.

which modification mainly accelerate FPS?

Thanks for your excellent work and your answering. I am quite interested that which part of modification mostly accelerate the FPS from about 2 to 5.
Looking forward to your answer.

How To train the model?

Dear HyeonseobNam:
I am learning your excellent job.Do you mind show me:How do you pretrain the network? What is your train set,validation set and test set ?

Best wishes,
Maiji

I have not NVidia GPU on my PC,what should i do to make your code run in a CPU Only Mode?

Downloading the VGG-M model

When I click the link to download the VGG-M model nothing happens?

PyTorch, plt.Axes function not work.

My PC spec. as follows.
Hardware: Xeon-E3 / 16G / GTX 650 /
Software: Ubuntu 16.04 / Anaconda 5.2(Python 2.7) / CUDA 9.0 / PyTorch 0.3.0

After I modified the size of opts['batch_test'] = 32
I can run following command.

$ python run_tracker.py -s DragonBaby

But when I want go further to do this:

$ python run_tracking.py -s DragonBaby -d -f

I got this information.

../modules/model.py:144: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
pos_loss = -F.log_softmax(pos_score)[:,1]
../modules/model.py:145: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
neg_loss = -F.log_softmax(neg_score)[:,0]
Traceback (most recent call last):
File "run_tracker.py", line 329, in
result, result_bb, fps = run_mdnet(img_list, init_bbox, gt=gt, savefig_dir=savefig_dir, display=display)
File "run_tracker.py", line 190, in run_mdnet
im = ax.imshow(image, aspect='normal')
File "/home/thwang/anaconda2/lib/python2.7/site-packages/matplotlib/init.py", line 1855, in inner
return func(ax, *args, **kwargs)
File "/home/thwang/anaconda2/lib/python2.7/site-packages/matplotlib/axes/_axes.py", line 5482, in imshow
self.set_aspect(aspect)
File "/home/thwang/anaconda2/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 1345, in set_aspect
aspect = float(aspect) # raise ValueError if necessary
ValueError: could not convert string to float: normal

I can't make this work properly.
Could you help me solve this?

code error in generate canditates when update

In line168 of run_tracker.py, but I think it should be:
neg_examples = neg_generator(target_bbox, opts['n_neg_update'], opts['overlap_neg_update'])
rather than:
neg_examples = neg_generator(target_bbox, opts['n_neg_update'], opts['overlap_neg_init'])

Missing ImageNet-VID pickle file

The file pretrain/data/imagenet_vid.pkl is missing from the repository. Any chance you can push it up? Thank you.

How long do you take training on ImageVid?

I want to know how long do you take training on ImageVid? thanks

Tracking Part

Hi all,
Before we are doing the tracking part, we must provide the bbox of groundtruth for each image. It seems like to generate the new bbox sample via original groundtruth bbox.
However, in the real case, when we are doing the tracking, we don't have the groundtruth information.
Is there any possible to track object without providing groundtruth bbox? To specific, how to generate samples and crop image without using groundtruth bbox values?

Thank you!

Change the CNN Model

Dear @HyeonseobNam,
Thank you for your fantastic work.
I want to change the Base CNN model of your code via another one (e.g., ResNet18). Do you have any idea that how can I do that?

cuda error (python run_tracker.py)

I got some issue with inference step. I think it's pytorch + cuda issue.
what's the problem?

python run_tracker.py -s DragonBaby -f

File "run_tracker.py", line 329, in
result, result_bb, fps = run_mdnet(img_list, init_bbox, gt=gt, savefig_dir=savefig_dir, display=display)
File "run_tracker.py", line 131, in run_mdnet
model = model.cuda()
File "/home/dhlee/anaconda2/envs/tensorflow/lib/python2.7/site-packages/torch/nn/modules/module.py", line 147, in cuda
return self._apply(lambda t: t.cuda(device_id))
File "/home/dhlee/anaconda2/envs/tensorflow/lib/python2.7/site-packages/torch/nn/modules/module.py", line 118, in _apply
module._apply(fn)
File "/home/dhlee/anaconda2/envs/tensorflow/lib/python2.7/site-packages/torch/nn/modules/module.py", line 118, in _apply
module._apply(fn)
File "/home/dhlee/anaconda2/envs/tensorflow/lib/python2.7/site-packages/torch/nn/modules/module.py", line 118, in _apply
module._apply(fn)
File "/home/dhlee/anaconda2/envs/tensorflow/lib/python2.7/site-packages/torch/nn/modules/module.py", line 124, in _apply
param.data = fn(param.data)
File "/home/dhlee/anaconda2/envs/tensorflow/lib/python2.7/site-packages/torch/nn/modules/module.py", line 147, in
return self._apply(lambda t: t.cuda(device_id))
File "/home/dhlee/anaconda2/envs/tensorflow/lib/python2.7/site-packages/torch/_utils.py", line 65, in cuda
return new_type(self.size()).copy(self, async)
File "/home/dhlee/anaconda2/envs/tensorflow/lib/python2.7/site-packages/torch/cuda/init.py", line 275, in new
return super(_CudaBase, cls).new(cls, *args, **kwargs)
KeyboardInterrupt

Why training in interpretation

Hi,

In the run_mdnet function in run_tracker.py, why do you train the box regressor and the model? Isn't the function is for interpretation using a pre-trained model?

Best regards,

Runtime Error: from negative region generation

Hi Hyeonseob,

I found py-MDNet very interesting and I would like to use it a lot.
So, I tried to train a new model based on a new dataset.

However, when I was running the program for the 4th cycle,
it returns "RuntimeError: the given numpy array has zero-sized dimensions. Zero-sized dimensions are not supported in PyTorch".

It is from data_prov.py on line 63
neg_regions = torch.from_numpy(neg_regions).float().

called by train_mdnet.py on line 64
pos_regions, neg_regions = dataset[k].next()

Thank you.
Best,
Peratham

Transpose the shape of the initial model weights

Hello Hyeonseob,

self.layers[i][0].weight.data = torch.from_numpy(np.transpose(weight, (3,2,0,1)))

Could you please explain to me, why did you transpose the shape of weights in the line
of code above, is there a difference between the weights of the initial model and the model
which you implemented in Pytorch ?

Thank you,

No such file or directory:

Request to Redistribution of py-MDNet

My name is Nina, the director of communication of the open source project Tegu. Tegu is a GUI-based software that allows developers to implement computer vision algorithms in their work with minimal amount of coding.

We have recently noticed your repo py-MDNet. What you wrote is very valuable and we appreciate your contribution to the open source community.

We think py-MDNet will provide greater benefits to the community if we are able to incorporate it as part of the growing Tegu project. Per your LICENSE file, we would therefore like to request your permission for redistribution and potential commercial use of the software.

Tegu will remain open source in the future and your work will be cited as directed in your LICENSE file.

Looking forward to hearing from you.

RuntimeError: CUDNN_STATUS_ARCH_MISMATCH

Traceback (most recent call last):
File "run_tracker.py", line 329, in
result, result_bb, fps = run_mdnet(img_list, init_bbox, gt=gt, savefig_dir=savefig_dir, display=display)
File "run_tracker.py", line 146, in run_mdnet
bbreg_feats = forward_samples(model, image, bbreg_examples)
File "run_tracker.py", line 34, in forward_samples
feat = model(regions, out_layer=out_layer)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "../modules/model.py", line 110, in forward
x = module(x)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.py", line 64, in forward
input = module(input)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 237, in forward
self.padding, self.dilation, self.groups)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/functional.py", line 40, in conv2d
return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_ARCH_MISMATCH

GPU: nvidia NVS 315
OS: Ubuntu 16.04

init_bbox for a sequence with multiple targets

If I have a sequence with more than one target, what should the initial bounding box be set to in gen_config?

where is lr_decay in the options_vot.yaml？

https://github.com/HyeonseobNam/py-MDNet/blob/28ff286ae0808f604690326684acb0a63ccf9b42/pretrain/train_mdnet.py#L40

where is the parameter lr_decay? Ican't find it。

pretrain

why it need pretrain step?

Why performance on TB100/TB50 worse than matlab repo(as paper display)?

I have noticed the performance figures in python repo are worse than these in matlab repo, Why?

mdnet_vot-otb.pth

how you get the mdnet_vot-otb.pth?
why in some case, the mdnet_vot-otb_new.pth is better than the mdnet_vot-otb.pth?

error in use"python run_tracker.py -d DragonBaby" and "python run_tracker.py -f DragonBaby"

wasson@wasson-MS-7971:~/yujia/py-MDNet-master/tracking$ python run_tracker.py -d DragonBaby
usage: run_tracker.py [-h] [-s SEQ] [-j JSON] [-f] [-d]
run_tracker.py: error: unrecognized arguments: DragonBaby

Running the program has encountered a problem

I use 'python run_tracker.py -s DragonBaby -d -f',but it is wrong.
This is the information printed by the terminal：
Traceback (most recent call last):
File "run_tracker.py", line 330, in
result, result_bb, fps = run_mdnet(img_list, init_bbox, gt=gt, savefig_dir=savefig_dir, display=display)
File "run_tracker.py", line 147, in run_mdnet
bbreg_feats = forward_samples(model, image, bbreg_examples)
File "run_tracker.py", line 35, in forward_samples
feat = model(regions, out_layer=out_layer)
File "/home/casic/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/usr/projects/ObjectTracking/single/py-MDNet-master/model.py", line 110, in forward
x = module(x)
File "/home/casic/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/cs/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/cs/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/usr/projects/ObjectTracking/single/py-MDNet-master/model.py", line 40, in forward
torch.cat((pad,pad,pad,pad,x_sq),2)),1)
RuntimeError: CUDA out of memory. Tried to allocate 1.24 GiB (GPU 0; 3.95 GiB total capacity; 1.77 GiB already allocated; 1.15 GiB free; 242.18 MiB cached)

do evaluation on OTB and VOT

Could you tell me how to do evaluation on OTB and VOT based on python?
Or could you publish the evaluation file?
I am learning tracking based on your architecture.
Thank you very much!

Segmentation fault (core dumped)

when I run python run_'tracker.py -s DragonBaby', it happened in line 36 of bbreg.py, 'self.model.fit(X, Y)' ,
python-->3.6.2
torch-->1.0.0
sklearn-->0.19.0
opencv-->3.1.0
GPU-->tesla p40 and have 22G empty

Problem during tracking

Hello, I encountered a problem when running "python tracking/run_tracker.py -s DragonBaby", the error below shows when I run the command:
Traceback (most recent call last):
File "tracking/run_tracker.py", line 325, in
result, result_bb, fps = run_mdnet(img_list, init_bbox, gt=gt, savefig_dir=savefig_dir, display=display)
File "tracking/run_tracker.py", line 150, in run_mdnet
train(model, criterion, init_optimizer, pos_feats, neg_feats, opts['maxiter_init'])
File "tracking/run_tracker.py", line 83, in train
score = model(batch_neg_feats[start:end], in_layer=in_layer)
File "C:\MIMIA\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File ".\modules\model.py", line 120, in forward
x = module(x)
File "C:\MIMIA\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\MIMIA\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\container.py", line 100, in forward
input = module(input)
File "C:\MIMIA\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\MIMIA\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "C:\MIMIA\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)

Please help me. Thanks!!!!

input image size during the training process

Are there any image size requirements during the training phase?

Support CPU Mode

Dear @HyeonseobNam,
How one can change your code to work on CPU mode (not using any GPU)?

Problem on training and tracking

Hello, if I want to run my own sequences, do I need to pretrain using my image datasets or I can just run tracking? Thank you!

Target score drops below 0 very fast

I'm f'tuning MDNet to my data. Only the first frame in the video is labelled with a bbox. After 8-10 frames the target score drop below 0. I played with learning rates/multipliers/trainable layers, but no success so far. Anything I could do or I need to label a larger number of frames in the dataset?

why it need train the fully connected layers for 10 iterations in online update?

why it need train the fully connected layers for 10 iterations in online update?

RuntimeError: CUDNN_STATUS_NOT_SUPPORTED. When run the train_mdnet.py

Traceback (most recent call last):
File "/home/kai_xu/PycharmProjects/py-MDNet/pretrain/train_mdnet.py", line 100, in
train_mdnet()
File "/home/kai_xu/PycharmProjects/py-MDNet/pretrain/train_mdnet.py", line 71, in train_mdnet
pos_score = model(pos_regions, k)
File "/home/kai_xu/anaconda3/envs/pytorch/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "../modules/model.py", line 111, in forward
x = module(x)
File "/home/kai_xu/anaconda3/envs/pytorch/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "/home/kai_xu/anaconda3/envs/pytorch/lib/python2.7/site-packages/torch/nn/modules/container.py", line 64, in forward
input = module(input)
File "/home/kai_xu/anaconda3/envs/pytorch/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "/home/kai_xu/anaconda3/envs/pytorch/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 237, in forward
self.padding, self.dilation, self.groups)
File "/home/kai_xu/anaconda3/envs/pytorch/lib/python2.7/site-packages/torch/nn/functional.py", line 40, in conv2d
return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

about the cfg.json

Hello.could you please tell me how to find the dataset include cfg.json

the success is different from what you show

hi,do you draw the otb100 success using the results "Raw results of MDNet pretrained on Imagenet-VID: Google drive link" Or you use the other results.Because when I draw the line,I only get at 0.6761(based on imagenet) and 0.6718(based on vot).THey are different from what you show 0.680(vid) and 0.676(vid).I used the otb pyhton to draw the success line https://github.com/jwlim/tracker_benchmark.thanks a lot

VOT

could you give the link of the VOT database?
I cannot download it
thanks

Improving timing issues

I have two questions:

1. Have you tried working in a fully convolution mode? I am thinking of changing it to a fully convolution,
   So it may run faster. Do you think there are any reasons for that no to work well?

2. I saw that in your PyTorch code, what's taking most of the time during inference is the cropping, and not the feed froward part.
   Have you tried addressing this issue? How do you think this can be resolved?

Cheers,
Avi

Copy previous result at failure

In 'run_tracker.py' , if the tracker lose the target in a frame(all sample scores are negative), you still update the target location. Shouldn't the target be locked at the previous location if all sample scores are negative?
if not success:
target_bbox = result[i-1]
bbreg_bbox = result_bb[i-1]
Thank you for you answer?

Model training randomness problem！

Hello,
when I used your model to carry out my work,
I found that it is difficult to verify the validity of the ingredients during the ablation experiment.
Due to the randomness of the model
how did you solve it?
Is it the average of multiple training tests, or is it a fixed random seed?
If it is the former, can you briefly describe how to draw the accuracy curve under the condition that multiple results are averaged.
I do n’t know much about this, just contact with target tracking,
Looking forward to your reply
best~