megvii-research / video_analyst

A series of basic algorithms that are useful for video understanding, including Single Object Tracking (SOT), Video Object Segmentation (VOS) and so on.

Video Analyst

This is the implementation of a series of basic algorithms which is useful for video understanding, including Single Object Tracking (SOT), Video Object Segmentation (VOS), etc.

Current implementation list:

• SOT
  • SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines [demo]

Example SiamFC++ outputs.

• VOS
  • State-Aware Tracker for Real-Time Video Object Segmentation

Example SAT outputs.

SOT Quick start

Setup

Please refer to SETUP.md, SOT_SETUP.md

Demo

SOT video demo

# demo with web camera
python3 ./demo/main/video/sot_video.py --config 'experiments/siamfcpp/test/vot/siamfcpp_alexnet.yaml' --device cuda --video "webcam" 

# demo with video file, and dump result into video file (optional)
python3 ./demo/main/video/sot_video.py --config 'experiments/siamfcpp/test/vot/siamfcpp_alexnet.yaml' --device cuda --video $video_dir/demo.mp4 --output $dump_path/result.mp4

# demo with extracted image files, and dump result into image files (optional)
python3 ./demo/main/video/sot_video.py --config 'experiments/siamfcpp/test/vot/siamfcpp_alexnet.yaml' --device cuda --video $video_dir/*.jpg --output $dump_dir

Test

Please refer to SOT_TEST.md for detail.

Training

Please refer to SOT_TRAINING.md for detail.

Repository structure (in progress)

project_root/
├── experiments  # experiment configurations, in yaml format
├── main
│   ├── train.py  # trainng entry point
│   └── test.py  # test entry point
├── video_analyst
│   ├── data  # modules related to data
│   │   ├── dataset  # data fetcher of each individual dataset
│   │   ├── sampler  # data sampler, including inner-dataset and intra-dataset sampling procedure
│   │   ├── dataloader.py  # data loading procedure
│   │   └── transformer  # data augmentation
│   ├── engine  # procedure controller, including traiing control / hp&model loading
│   │   ├── monitor  # monitor for tasks during training, including visualization / logging / benchmarking
│   │   ├── trainer.py  # train a epoch
│   │   ├── tester.py  # test a model on a benchmark
│   ├── model # model builder
│   │   ├── backbone  # backbone network builder
│   │   ├── common_opr  # shared operator (e.g. cross-correlation)
│   │   ├── task_model  # holistic model builder
│   │   ├── task_head  # head network builder
│   │   └── loss  # loss builder
│   ├── pipeline  # pipeline builder (tracking / vos)
│   │   ├── segmenter  # segmenter builder for vos
│   │   ├── tracker  # tracker builder for tracking
│   │   └── utils  # pipeline utils
│   ├── config  # configuration manager
│   ├── evaluation  # benchmark
│   ├── optim  # optimization-related module (learning rate, gradient clipping, etc.)
│   │   ├── optimizer # optimizer
│   │   ├── scheduler # learning rate scheduler
│   │   └── grad_modifier # gradient-related operation (parameter freezing)
│   └── utils  # useful tools
└── README.md

docs

For detail, please refer to markdown files under docs.

SOT

• SOT_SETUP.md: instructions for setting-up
• SOT_MODEL_ZOO.md: description of released sot models
• SOT_TRAINING.md: details related to training
• SOT_TEST.md: details related to test

VOS

• VOS_SETUP.md: instructions for setting-up
• VOS_MODEL_ZOO.md: description of released sot models
• VOS_TRAINING.md: details related to training
• VOS_TEST.md: details related to training

DEVELOP

• DEVELOP.md: description of project design (registry, configuration tree, etc.)
• PIPELINE_API.md: description for pipeline API
• FORMATTING_INSTRUCTION: instruction for code formatting (yapf/isort/flake/etc.)

TODO

[] refine code stype and test cases

Acknowledgement

• video_analyst/evaluation/vot_benchmark and other related code have been borrowed from PySOT
• video_analyst/evaluation/got_benchmark and other related code have been borrowed from got-toolkit
• detectron2
• fvcore
• pytracking
• DROL

References

@inproceedings{xu2020siamfc++,
  title={SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines.},
  author={Xu, Yinda and Wang, Zeyu and Li, Zuoxin and Yuan, Ye and Yu, Gang},
  booktitle={AAAI},
  pages={12549--12556},
  year={2020}
}

@inproceedings{chen2020state,
  title={State-Aware Tracker for Real-Time Video Object Segmentation},
  author={Chen, Xi and Li, Zuoxin and Yuan, Ye and Yu, Gang and Shen, Jianxin and Qi, Donglian},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9384--9393},
  year={2020}
}

Contact

Maintainer (sorted by family name):

• Xi Chen@XavierCHEN34
• Zuoxin Li@lzx1413
• Zeyu Wang@JWarlock
• Yinda Xu@MARMOTatZJU

video_analyst's Issues

Has the "config example" setting the dataset soft link in SOT been deleted?

Has the "config example" setting the dataset soft link in single target tracking been deleted?

OTB Test Code

Hi, I am trying to implement test code for OTB100, I basically copy and paste tester_impl/lasot.py to create otb.py. However, when I tried to run the test, it gives the following error:

 File "main/test.py", line 36, in <module>
    root_cfg.merge_from_file(exp_cfg_path)
  File "/home/anaconda3/envs/siamfc++/lib/python3.6/site-packages/yacs/config.py", line 213, in merge_from_file
    self.merge_from_other_cfg(cfg)
  File "/home/anaconda3/envs/siamfc++/lib/python3.6/site-packages/yacs/config.py", line 217, in merge_from_other_cfg
    _merge_a_into_b(cfg_other, self, self, [])
  File "/home/anaconda3/envs/siamfc++/lib/python3.6/site-packages/yacs/config.py", line 460, in _merge_a_into_b
    _merge_a_into_b(v, b[k], root, key_list + [k])
  File "/home/anaconda3/envs/siamfc++/lib/python3.6/site-packages/yacs/config.py", line 460, in _merge_a_into_b
    _merge_a_into_b(v, b[k], root, key_list + [k])
  File "/home/anaconda3/envs/siamfc++/lib/python3.6/site-packages/yacs/config.py", line 460, in _merge_a_into_b
    _merge_a_into_b(v, b[k], root, key_list + [k])
  [Previous line repeated 1 more time]
  File "/home/anaconda3/envs/siamfc++/lib/python3.6/site-packages/yacs/config.py", line 473, in _merge_a_into_b
    raise KeyError("Non-existent config key: {}".format(full_key))
KeyError: 'Non-existent config key: test.track.tester.OTBTester.dataset_names'

Any ideas on why it might be?

result problem

I use the your training code to train. The training set only uses got10k. The test results are as follows
my result:AO=0.612，SR0.5=0.735,SR0.75=0.461
your result:AO=0.595,SR0.5=0.695,SR0.75=0.479
Looks like performance is very good
But when I tested vot18 with this model, the result was very poor,The test results are as follows
my result:ACC=0.549,ROB=0.300,EAO=0.310
your result:ACC=0.587,ROB=0.183,EAO=0.426
Do you have this problem

It is worth noting that the code for training is "sh tools/train_test-googlenet.sh" . When training is interrupted, I run "python ./main/train.py --config experiments/siamfcpp/train/siamfcpp_googlenet-trn.yaml --resume-from-file=... " to continue training

TrackingNet baiduyunpan link?

Sorry to bother you again, can you provide TrackingNet datasets baiduyunpan download link? Because I haven't downloaded it successfully.

Test VOT

When I test my trained models using VOT datasets, I always run into the following problems.

(vincent) ubuntu@ubun:~/Vincent/object_track/video_analyst-master$ python ./main/test.py --config 'experiments/siamfcpp/train/lasot/siamfcpp_googlenet-trn.yaml'
2020-07-22 17:03:57.370 | INFO     | __main__:<module>:65 - Load experiment configuration at: /home/ubuntu/Vincent/object_track/video_analyst-master/experiments/siamfcpp/train/lasot/siamfcpp_googlenet-trn.yaml
2020-07-22 17:03:57.764 | INFO     | videoanalyst.model.module_base:update_params:62 - Load pretrained SiamTrack parameters from: /home/ubuntu/Vincent/object_track/video_analyst-master/snapshots/siamfcpp_googlenet-lasot/epoch-19.pkl whose md5sum is fc8c27d28ac7585f2a4c9b7a848a6767
2020-07-22 17:03:57.768 | INFO     | videoanalyst.engine.tester.tester_impl.vot:run_tracker:98 - runing test on devices [device(type='cuda', index=0)]
2020-07-22 17:03:57.768 | INFO     | videoanalyst.engine.tester.tester_impl.vot:run_tracker:100 - Using dataset VOT2018 at: /home/ubuntu/Vincent/object_track/video_analyst-master/datasets/VOT/vot2018
  0%|                                                                                                                                                                                | 0/60 [00:00<?, ?it/s]Traceback (most recent call last):
  File "./main/test.py", line 80, in <module>
    tester.test()
  File "/home/ubuntu/Vincent/object_track/video_analyst-master/videoanalyst/engine/tester/tester_impl/vot.py", line 87, in test
    self.run_tracker()
  File "/home/ubuntu/Vincent/object_track/video_analyst-master/videoanalyst/engine/tester/tester_impl/vot.py", line 115, in run_tracker
    self.worker(keys, all_devs[0], result_queue, speed_queue)
  File "/home/ubuntu/Vincent/object_track/video_analyst-master/videoanalyst/engine/tester/tester_impl/vot.py", line 168, in worker
    lost, speed = self.track_single_video(tracker, video, v_id=v_id)
  File "/home/ubuntu/Vincent/object_track/video_analyst-master/videoanalyst/engine/tester/tester_impl/vot.py", line 232, in track_single_video
    package="vot_overlap").vot_overlap
  File "/home/ubuntu/anaconda3/envs/vincent/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/Vincent/object_track/video_analyst-master/videoanalyst/evaluation/vot_benchmark/pysot/utils/__init__.py", line 9, in <module>
    from . import region
ImportError: cannot import name 'region' from 'videoanalyst.evaluation.vot_benchmark.pysot.utils' (/home/ubuntu/Vincent/object_track/video_analyst-master/videoanalyst/evaluation/vot_benchmark/pysot/utils/__init__.py)
  0%|                                                                                                                                                                                | 0/60 [00:02<?, ?it/s]

Then , I found the file structure in the 'video_analyst-master/videoanalyst/evaluation/vot_benchmark/pysot/utils/' directory looks like this：

(vincent) ubuntu@ubun:~/Vincent/object_track/video_analyst-master/videoanalyst/evaluation/vot_benchmark/pysot/utils$ ls
c_region.pxd  __init__.py  misc.py  __pycache__  region.pyx  setup.py  src  statistics.py

This '. pyx 'file causes the direct use of the from . import region in __init__.py to fail

Training interruption

When I train this model, the training process will be interrupted without any reason, but there is no error message. I don't know if you have this problem during your training. I haven't changed the training code, and the environment is configured according to README.

YouTube-BB 数据集

请问您有处理好的YouTube-BB 数据集的百度网盘么，从官方下载视频文件实在太大了，梯子的流量不够。

Result on VOT2018

I trained the network according to paper and got AUC 56.2% on LaSOT dataset. But the EAO on the VOT2018 dataset only 0.31. I don't know the reason. Is there something special I need to notice? The setting of the training env:
Pytorch1.1.0, CUDA10.0,
GPU：4*2080Ti
use fulldata.
Besides, the result of using the vot evaluation of this code is lower than using official code（i.e., matlab code） by 0.07. For example, when I test the VOT dataset using the model you offered, the evaluation of this code gives 0.35 EAO, but if I put the tested result into the official evaluation code, the result is 0.42 which is similar to the paper. Why?
Expect for your explanation, thanks~

非常感谢您的工作，想问下，训练部分什么时候放出来呢

dataset

Hello, if I use the got-10k data set provided by the official website, does the dataset need to be preprocessed before training? (for example, siamrpn + +, I need to do pre_crop 511 12 before training). If I want to add other training datasets, such as coco vid, is the configuration file of the datasets the same as that of got10K.py？Thank you very much for your answer

about training strategy

Hello, I want to ask two questions:

1. It seems the training strategy in this repo yaml isn't exactly same as that in paper, maybe be the difference between internal framework and PyTorch, so you choose the better performance cfg for us in this repo yaml file?
2. Does the yaml file whose name doesn't include 'dist' prepare for single GPU machine training? And if I have a machine which just has single GPU and less CPU threads, can you give some suggestions where should I modify in the cfg file?
   Sincerely appreciate for any help!!

About the size of the search area during testing

I noticed that during training, the search area calculation method of x frame and z frame is similar to siamfc, but when testing, they appear different. This difference comes from the get_crop function (https://github.com/MegviiDetection/video_analyst/blob/b786f48e92d6cd3d2759d55372e8c8ef76f6b5e5/videoanalyst/pipeline/utils/crop.py#L108).
In the z frame during the test, since the x_size parameter is not passed to the get_crop function, the calculation of the search area of the z frame is similar to siamfc.
https://github.com/MegviiDetection/video_analyst/blob/b786f48e92d6cd3d2759d55372e8c8ef76f6b5e5/videoanalyst/pipeline/tracker_impl/siamfcpp_lt_track.py#L155
In the x frame during the test, since the x_size parameter is passed to the get_crop function, the search area of the x frame is calculated as x_size/(z_size / s_crop).
https://github.com/MegviiDetection/video_analyst/blob/b786f48e92d6cd3d2759d55372e8c8ef76f6b5e5/videoanalyst/pipeline/tracker_impl/siamfcpp_lt_track.py#L235

Then there is a problem:
During training, the size of the search area of the x and z frames is only related to the target size, and not related to the size of x_size and z_size.
In the test, the size of the search area of the x and z frames is not only related to the target size, but also related to the size of x_size and z_size. This causes the search area of the x frame to be much larger than the z frame.

I think more contextual information will affect the performance of the tracker.
So I want to ask, does increasing the search area affect the performance of the tracker during testing? Or what misunderstanding I have, I hope you can correct my misunderstanding.
Thanks~

Designate GPU computing

Can you tell me how to designate certain GPUs on a computer for calculation？

about setup

when i obey the guidelines in SOT_SETUP.md, after i conda activate siamfcpp and i surely in the conda virtual environment, and after i ran pip3 install -U -r requirements.txt，some packages always can't be installed successfully(although in terminal it shows they are installed successfully) beacause i import these packages, they return ModuleNotFoundError, so where are wrong?

Where can we find the tracking results of siamfc++?

Have you released tracking results of siamfc++ and where can we find them? Thanks for your great work.

can not train

when i run sh tools/train_test-googlent.sh, An error occurred：

Where should I download these files？

tracking results by using fulldata

请问在使用fulldata之后在OTB100，VOT2018，VOT2019上面表现如何

Using first frame

SiamFC + + does not seem to use the data of the first frame to fine tune the model. What is the reason? Is it because there is enough training data, so the effect of fine-tuning is not enough?

Result problem about MODEL_ZOO

I. Question
I used the pre-trained models from MODEL_ZOO for test on VOT2018, but got lower EAO than scores below.

I wonder how I can fix that. Thank you!

II. Useful information
1. Here is my EAO scores
(1) torch1.4.0 ( with default cudatoolkit installed by conda)


(2) torch1.1.0( with default cudatoolkit installed by conda)

2. Here is my version of package( output of 'pip freeze' )
(1) for torch1.4.0
absl-py==0.9.0
aiocontextvars==0.2.2
astroid==2.4.2
backcall==0.2.0
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
colorama==0.3.7
contextvars==2.4
cycler==0.10.0
Cython==0.27.3
decorator==4.4.2
future==0.18.2
google-auth==1.18.0
google-auth-oauthlib==0.4.1
grpcio==1.30.0
idna==2.10
imageio==2.9.0
immutables==0.14
importlib-metadata==1.7.0
ipython==7.16.1
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.17.1
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
llvmlite==0.32.1
loguru==0.4.1
Markdown==3.2.2
matplotlib==3.2.2
mccabe==0.6.1
networkx==2.4
numba==0.45.0
numpy==1.19.0
oauthlib==3.1.0
opencv-python==4.3.0.36
pandas==0.25.3
parso==0.7.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
prompt-toolkit==3.0.3
protobuf==3.12.2
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.1
Pygments==2.6.1
pylint==2.5.3
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
PyWavelets==1.1.1
PyYAML==5.3.1
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-image==0.17.2
scipy==1.2.1
Shapely==1.7.0
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tifffile==2020.7.4
toml==0.10.1
torch==1.4.0
torchvision==0.5.0
tqdm==4.47.0
traitlets==4.3.3
typed-ast==1.4.1
urllib3==1.25.9
wcwidth==0.2.5
Werkzeug==1.0.1
wget==3.2
wrapt==1.12.1
yacs==0.1.7
yapf==0.28.0
zipp==3.1.0
(2) for torch1.1.0
absl-py==0.9.0
aiocontextvars==0.2.2
astroid==2.4.2
backcall==0.2.0
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
colorama==0.3.7
contextvars==2.4
cycler==0.10.0
Cython==0.27.3
decorator==4.4.2
future==0.18.2
google-auth==1.18.0
google-auth-oauthlib==0.4.1
grpcio==1.30.0
idna==2.10
imageio==2.9.0
immutables==0.14
importlib-metadata==1.7.0
ipython==7.16.1
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.17.1
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
llvmlite==0.32.1
loguru==0.4.1
Markdown==3.2.2
matplotlib==3.2.2
mccabe==0.6.1
networkx==2.4
numba==0.45.0
numpy==1.19.0
oauthlib==3.1.0
opencv-python==4.3.0.36
pandas==1.0.5
parso==0.7.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
prompt-toolkit==3.0.5
protobuf==3.12.2
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.1
Pygments==2.6.1
pylint==2.5.3
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
PyWavelets==1.1.1
PyYAML==5.3.1
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-image==0.17.2
scipy==1.2.1
Shapely==1.7.0
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tifffile==2020.7.4
toml==0.10.1
torch==1.1.0
torchvision==0.3.0
tqdm==4.47.0
traitlets==4.3.3
typed-ast==1.4.1
urllib3==1.25.9
wcwidth==0.2.5
Werkzeug==1.0.1
wget==3.2
wrapt==1.12.1
yacs==0.1.7
yapf==0.28.0
zipp==3.1.0

Numba error during evaluation

Hi,
I am trying to evaluate the existing models on VOT 2018 and VOT2019. I get the metrics but after that immediately it throws the following error

Traceback (most recent call last):

File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/errors.py", line 662, in new_error_context
yield
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 258, in lower_block
self.lower_inst(inst)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 301, in lower_inst
val = self.lower_assign(ty, inst)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 459, in lower_assign
return self.lower_expr(ty, value)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 919, in lower_expr
res = self.lower_call(resty, expr)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 711, in lower_call
res = self._lower_call_normal(fnty, expr, signature)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 890, in _lower_call_normal
res = impl(self.builder, argvals, self.loc)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/targets/base.py", line 1132, in call
res = self._imp(self._context, builder, self._sig, args, loc=loc)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/targets/base.py", line 1157, in wrapper
return fn(*args, **kwargs)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/targets/arrayobj.py", line 3375, in numpy_zeros_nd
_zero_fill_array(context, builder, ary)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/targets/arrayobj.py", line 3305, in _zero_fill_array
cgutils.memset(builder, ary.data, builder.mul(ary.itemsize, ary.nitems), 0)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/cgutils.py", line 866, in memset
builder.call(fn, [ptr, value, size, int32_t(0), bool_t(0)])
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/llvmlite/ir/builder.py", line 841, in call
cconv=cconv, tail=tail, fastmath=fastmath)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/llvmlite/ir/instructions.py", line 84, in init
raise TypeError(msg)
TypeError: Type of #4 arg mismatch: i1 != i32

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./main/test.py", line 51, in
tester.test()
File "/home/meetkirankum/AdaTrack/video_analyst/videoanalyst/engine/tester/tester_impl/vot.py", line 91, in test
self.evaluation()
File "/home/meetkirankum/AdaTrack/video_analyst/videoanalyst/engine/tester/tester_impl/vot.py", line 197, in evaluation
ret = benchmark.eval(self.tracker_name)
File "/home/meetkirankum/AdaTrack/video_analyst/videoanalyst/evaluation/vot_benchmark/pysot/evaluation/eao_benchmark.py", line 53, in eval
eao = self._calculate_eao(tracker_name, self.tags)
File "/home/meetkirankum/AdaTrack/video_analyst/videoanalyst/evaluation/vot_benchmark/pysot/evaluation/eao_benchmark.py", line 245, in _calculate_eao
expected_overlaps = calculate_expected_overlap(fragments, fweights)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/dispatcher.py", line 395, in _compile_for_args
raise e
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/dispatcher.py", line 352, in _compile_for_args
return self.compile(tuple(argtypes))
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/dispatcher.py", line 693, in compile
cres = self._compiler.compile(args, return_type)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/dispatcher.py", line 76, in compile
status, retval = self._compile_cached(args, return_type)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/dispatcher.py", line 90, in _compile_cached
retval = self._compile_core(args, return_type)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/dispatcher.py", line 108, in _compile_core
pipeline_class=self.pipeline_class)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 972, in compile_extra
return pipeline.compile_extra(func)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 390, in compile_extra
return self._compile_bytecode()
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 903, in _compile_bytecode
return self._compile_core()
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 890, in _compile_core
res = pm.run(self.status)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 266, in run
raise patched_exception
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 257, in run
stage()
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 764, in stage_nopython_backend
self._backend(lowerfn, objectmode=False)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 703, in _backend
lowered = lowerfn()
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 690, in backend_nopython_mode
self.metadata)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/compiler.py", line 1143, in native_lowering_stage
lower.lower()
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 177, in lower
self.lower_normal_function(self.fndesc)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 218, in lower_normal_function
entry_block_tail = self.lower_function_body()
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 243, in lower_function_body
self.lower_block(block)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/lowering.py", line 258, in lower_block
self.lower_inst(inst)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/errors.py", line 670, in new_error_context
six.reraise(type(newerr), newerr, tb)
File "/home/meetkirankum/anaconda3/envs/SOT/lib/python3.6/site-packages/numba/six.py", line 659, in reraise
raise value
numba.errors.LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)
Type of #4 arg mismatch: i1 != i32

File "videoanalyst/evaluation/vot_benchmark/pysot/utils/statistics.py", line 166:
def calculate_expected_overlap(fragments, fweights):

max_len = fragments.shape[1]
expected_overlaps = np.zeros((max_len), np.float32)
^

[1] During: lowering "$0.10 = call $0.6(max_len, $0.9, func=$0.6, args=[Var(max_len, /home/meetkirankum/AdaTrack/video_analyst/videoanalyst/evaluation/vot_benchmark/pysot/utils/statistics.py (165)), Var($0.9, /home/meetkirankum/AdaTrack/video_analyst/videoanalyst/evaluation/vot_benchmark/pysot/utils/statistics.py (166))], kws=(), vararg=None)" at /home/meetkirankum/AdaTrack/video_analyst/videoanalyst/evaluation/vot_benchmark/pysot/utils/statistics.py (166)

training problems

请问full data你们是用多少张GPU训练多少小时

training log

Can u please provide the training log of Google-Net Got10K version ? thank u please

unzip trackingNet

Thank you for providing the link to download the dataset. But I have some questions, how should I extract files from TEST.zip.001？Should I merge these files (TEST.zip.001/TEST.zip.002...) before unzip?

run 'siamfcpp_googlenet_votlt.yaml' and how to get classification scores

I can run Python 3. / main successfully/ test.py --config experiments/siamfcpp/test/vot/siamfcpp_ googlenet.yaml '
However, I can't run Python 3. / main successfully/ test.py --config'experiences / siamfcpp / test / VOT / siamfcpp_ googlenet_ votlt.yaml '.The error is 'No such file or directory: '/SiamFC++/datasets/VOT2018LT/ list.txt. How to place vot2018LT.
Besides, if I want to get the classification scores in every video of VOT2018 dataset, what can i do?Thank you very much.

TrackingNet datasets?

I also need trackingnet datasets, but I always make mistakes when I download them. Can you provide download links like Baidu cloud drives?

SOT_SETUP.md

sorry!your SOT_SETUP.md is 404,can you upload a new SOT_SETUP.md

SAT Train Problem

 I sat_res50_davis17.yaml as config file, and I just want to train on the Davis dataset. So i delete COCO and YoutubeVOS in sat_res50_davis17.yaml.  But when i run python3 ./main/dist_train_sat.py --config 'experiments/sat/train/sat_res50_davis17.yaml',it's wrong.

Traceback (most recent call last):
File "./main/dist_train_sat.py", line 152, in
root_cfg.merge_from_file(exp_cfg_path)
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/yacs/config.py", line 213, in merge_from_file
self.merge_from_other_cfg(cfg)
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/yacs/config.py", line 217, in merge_from_other_cfg
_merge_a_into_b(cfg_other, self, self, [])
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/yacs/config.py", line 460, in _merge_a_into_b
_merge_a_into_b(v, b[k], root, key_list + [k])
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/yacs/config.py", line 460, in _merge_a_into_b
_merge_a_into_b(v, b[k], root, key_list + [k])
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/yacs/config.py", line 473, in _merge_a_into_b
raise KeyError("Non-existent config key: {}".format(full_key))
KeyError: 'Non-existent config key: train.vos.tracker'
I guess there's something wrong with sat_res50_davis17.yaml. Looking forward to your reply,thanks

SAT Test Problem

When I use python3 ./main/test.py --config 'experiments/sat/test/sat_res50-davis17.yaml' to test model. Finally,it output the J&F-Mean, But when i check the mask,It's just a black background,like this:

Feature map of network output and target seem not match in tracking

Take Alexnet for example, As far as I know, the network take 303✖️303 search image as input and do a few downsampling and finally output 17✖️17 feature map of classification score and center score (fcos_cls_score_final in code). So the 17✖️17 matches the 303✖️303. Even considering some frontier info loss caused by no padding, the 17✖️17 will still match maybe some area like 200✖️200 in my opinion.
However when it comes to target generation in 'make_densebox_target.py', we just crop a (17✖️8)✖️(17✖️8)=(136✖️136) area from the center of 303✖️303 and use it to calculate target. This area seems too small, and doesn't match the network output.
In other words, 17*17 seems to match a much larger area of input search image in network, and match a very small area in target calculating. Which in my opinion is not reasonable.

Maybe I misunderstand some code, welcome to point out.

超参搜索

请问是否考虑写一个超参搜索函数，类似于pysot里面的hp_search.py

some question in paper

What is the label of the cls branch? I can't find the defined cls, Is the region of bbox mapped to 1 in the featuremap ? And how to defined the regession branch label ?
In the SiamRPN++,I know the label is defined by cal the iou between anchor and gt. but I can't find the anchor free style model that how to define the label .
Give you my best wishes,bro,I wanna to get the answer.

about training?

Excuse me, is it just showing the training code on the GOT-10K dataset? not ILSVRC-VID/DET_VID, Youtube-BB etc.

resuming bug with optimizer under DDP

Error [email protected]()
May due to state parameter location.

some advice about sot video demo

Hello @MARMOTatZJU @lzx1413 Thanks for your wonderful work and open codes.I have try to use sot video demo(demo/main/video/sot_video.py) and give you some feedback so that you can improve or fix them:

• in line66, it should be ''dump_only" not "dump-only";
• After line157 it should add another line:cv2.waitKey(1) so the track result frame will be showed correctly, otherwise it's just a white window;
• For third demo way(that is, with extracted image files), don't forget to add "-i " params to define object location in initial frame and when use param "--video", don't forget to add quotation marks to path;
• the last is that for test image whose size is less than 128×128, the template can't be displayed on the left-top of current frame. If in future work it had better use proportionate size not fixed size.
  Above are my advice. Thanks again!

cuda version

which version of CUDA does the author use?
When I use CUDA10.1,2080ti,torch==1.4.0torchvision==0.5.0,
it will get the following error：
CUDNN_STATUS_EXECUTION_FAILED

The code about SAT

Hi, @lzx1413, this is great project! Thank you so much! By the way, When will you open the source code about SAT? I am very interesting in this amazing job. I'm looking forward to seeing the code soon

When do you public shufflenetv2x0.5, shufflenetv2x1.0 results on VOT2018?

Thank you in advance!
We trained shufflenetv2x0.5, shufflenetv2x1.0 on VOT2018, but the results are not consistent with our cognition, So can you provide your results as a reference?

SiamFCpp tracker pretrained model download link

https://github.com/MegviiDetection/video_analyst/blob/master/docs/TUTORIALS/VOS_SETUP.md
SiamFCpp tracker download is not stable on Weiyun. Could you provide a Google drive link?

Both train-set and test-set in LaSOT are used during the training process.

https://github.com/MegviiDetection/video_analyst/blob/141f54d34cf6ceac59f6679bc101c4879271c196/experiments/siamfcpp/train/fulldata/siamfcpp_googlenet-trn-fulldata.yaml#L120

I notice a mistake. The above setting will make both train-set and test-set in LaSOT are used during the training process. This will lead a 59.0% even higher in AUC in LaSOT.
This should change to ' subset: "train" '

There is a question about the calculation of LOSS in SOT training

I found that the mask of ctr loss is a non-background area in any case. But in my opinion, if it is a neg pair, there is no need to calculate the ctr loss of the non-background area.
What do you think?

Link to models in BaiduYun invalid

Hi, I am trying to download models from the Baidu Yun Link to finish the last step of setup, it seems the provided Baidu Yun Link is invalid?

RuntimeError: Shared memory manager connection has timed out at /tmp/pip-req-build-ufslq_a9/torch/lib/libshm/core.cpp:99

❓ Questions and Help

when I train on the default config, after a few epochs, I run into this error

epoch 5, lr: 7.1e-02, cls: 0.089, ctr: 0.030, reg: 0.889, iou: 0.756, data: 7.5e-04, fwd: 3.0e-01, bwd: 1.5e-01, optimepoch 5, lr: 7.1e-02, cls: 0.089, ctr: 0.030, reg: 0.889, iou: 0.756, data: 7.5e-04, fwd: 3.0e-01, bwd: 1.5e-01, optim
epoch 5, lr: 7.1e-02, cls: 0.093, ctr: 0.030, reg: 0.877, iou: 0.759, data: 1.9e+01, fwd: 1.3e+00, bwd: 2.5e-01, optimepoch 5, lr: 7.1e-02, cls: 0.093, ctr: 0.030, reg: 0.877, iou: 0.759, data: 1.9e+01, fwd: 1.3e+00, bwd: 2.5e-01, optim
epoch 5, lr: 7.1e-02, cls: 0.090, ctr: 0.031, reg: 0.942, iou: 0.745, data: 1.2e-04, fwd: 9.9e-01, bwd: 1.5e-01, optimepoch 5, lr: 7.1e-02, cls: 0.090, ctr: 0.031, reg: 0.942, iou: 0.745, data: 1.2e-04, fwd: 9.9e-01, bwd: 1.5e-01, optim
epoch 5, lr: 7.1e-02, cls: 0.093, ctr: 0.031, reg: 0.887, iou: 0.757, data: 1.2e-04, fwd: 3.0e-01, bwd: 1.5e-01, optimepoch 5, lr: 7.1e-02, cls: 0.093, ctr: 0.031, reg: 0.887, iou: 0.757, data: 1.2e-04, fwd: 3.0e-01, bwd: 1.5e-01, optim
epoch 5, lr: 7.1e-02, cls: 0.098, ctr: 0.035, reg: 0.886, iou: 0.756, data: 1.1e-04, fwd: 3.0e-01, bwd: 1.5e-01, optimepoch 5, lr: 7.1e-02, cls: 0.098, ctr: 0.035, reg: 0.886, iou: 0.756, data: 1.1e-04, fwd: 3.0e-01, bwd: 1.5e-01, optim
epoch 5, lr: 7.1e-02, cls: 0.100, ctr: 0.033, reg: 0.897, iou: 0.754, data: 9.2e-05, fwd: 3.0e-01, bwd: 1.5e-01, optimepoch 5, lr: 7.1e-02, cls: 0.100, ctr: 0.033, reg: 0.897, iou: 0.754, data: 9.2e-05, fwd: 3.0e-01, bwd: 1.5e-01, optim
: 2.2e-01, : 18%|█▊ | 106/585 [04:24<20:47, 2.60s/it]Traceback (most recent call last):
File "./main/train.py", line 111, in
trainer.train()
File "/mnt/lustre/zhaowenqian/tracking/video_analyst-master/videoanalyst/engine/trainer/trainer_impl/regular_trainer.py", line 100, in train
training_data = next(self._dataloader)
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in next
data = self._next_data()
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 838, in _next_data
return self._process_data(data)
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
data.reraise()
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 7.
Original Traceback (most recent call last):
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 74, in
return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 53, in default_collate
storage = elem.storage()._new_shared(numel)
File "/mnt/lustre/zhaowenqian/anaconda3/lib/python3.7/site-packages/torch/storage.py", line 128, in _new_shared
return cls._new_using_filename(size)
RuntimeError: Shared memory manager connection has timed out at /tmp/pip-req-build-ufslq_a9/torch/lib/libshm/core.cpp:99

RuntimeError: Shared memory manager connection has timed out at /tmp/pip-req-build-ufslq_a9/torch/lib/libshm/core.cpp:99

which I cannot locate where the problem is

Does experiments folder contain VOS related experiments?

From the file names, they don't look like VOS related. Could you confirm?

reproduce results

Can we directly reproduce the results in paper without any further parameter fine-tune?

CUDA out of memory

Hi I am trying to run DDP training, it seems 4 gpu is not enough for my hardware device. However it seems not working when I set os.environ[“CUDA_VISIBLE_DEVICES”]=“0,1,2,3,4,5,6,7” as it’s still training on 4 gpu. I wonder why that might be? Thanks!

Why not sample negative examples in batch?

Sampling negative pairs from images in the same batch and do some hard negative example mining is a common trick in image retrieval tasks. However in siamese network I didn't find anyone use it, they all make negative pairs only in dataloader. I have tried sample different pairs in batch as negative example and then sort them to get hard negative examples, but it doesn't work very well. So are there some specific reasons that we don't use this trick in tracking tasks, or just no one would like to try?

sh tools/sat/test/test_DAVIS2017.sh error

when I run the code ‘sh tools/sat/test/test_DAVIS2017.sh’ ,then there would be an error:
2020-05-19 20:45:57.972 | INFO | main::65 - Load experiment configuration at: /home/work/guoxianda/video_analyst/experiments/sat/test/sat_res50-davis17.yaml
2020-05-19 20:45:58.291 | INFO | videoanalyst.model.module_base:update_params:62 - Load pretrained SiamTrack parameters from: /home/work/guoxianda/video_analyst/models/siamfcpp/siamfcpp-alexnet-vot-md5_18fd31a2f94b0296c08fff9b0f9ad240.pkl whose md5sum is 18fd31a2f94b0296c08fff9b0f9ad240
2020-05-19 20:46:16.195 | INFO | videoanalyst.model.module_base:update_params:62 - Load pretrained SatVOS parameters from: /home/work/guoxianda/video_analyst/models/sat/sat_res50_davis17_b81fa.pkl whose md5sum is 61900ceacf73cc10fcf242491cbb81fa
2020-05-19 20:46:16.204 | INFO | videoanalyst.engine.tester.tester_impl.davis:run_tracker:94 - runing test on devices [device(type='cuda', index=0), device(type='cuda', index=1), device(type='cuda', index=2), device(type='cuda', index=3)]
2020-05-19 20:46:16.204 | INFO | videoanalyst.engine.tester.tester_impl.davis:run_tracker:96 - Using dataset DAVIS2017 at: /home/work/guoxianda/video_analyst/datasets/DAVIS
0%| | 0/30 [00:00<?, ?it/s]2020-05-19 20:46:17.498 | INFO | videoanalyst.engine.tester.tester_impl.davis:run_tracker:125 - process:0, start:0, end:8
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "./main/test.py", line 80, in
tester.test()
File "/home/work/guoxianda/video_analyst/videoanalyst/engine/tester/tester_impl/davis.py", line 83, in test
self.run_tracker()
File "/home/work/guoxianda/video_analyst/videoanalyst/engine/tester/tester_impl/davis.py", line 126, in run_tracker
proc.start()
File "/home/work/anaconda3/envs/venv3/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/home/work/anaconda3/envs/venv3/lib/python3.6/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/home/work/anaconda3/envs/venv3/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/home/work/anaconda3/envs/venv3/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/home/work/anaconda3/envs/venv3/lib/python3.6/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/home/work/anaconda3/envs/venv3/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/home/work/anaconda3/envs/venv3/lib/python3.6/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "/home/work/anaconda3/envs/venv3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 324, in reduce_storage
metadata = storage.share_filename()
RuntimeError: error executing torch_shm_manager at "/home/work/anaconda3/envs/venv3/lib/python3.6/site-packages/torch/bin/torch_shm_manager" at /pytorch/torch/lib/libshm/core.cpp:99

Any solutions to this? thank you very much! looking forward to your reply!

raise StopIteration

在训练的过程中，有这样的问题：

2020-07-19 19:56:00.637 | INFO     | videoanalyst.engine.monitor.monitor_impl.tensorboard_logger:update:75 - Tensorboard writer built, starts recording from global_step=0
2020-07-19 19:56:00.637 | INFO     | videoanalyst.engine.monitor.monitor_impl.tensorboard_logger:update:78 - epoch=0, max_epoch=20, iteration=0, max_iteration=2343
epoch 0, lr: 8.0e-02, cls: 0.106, ctr: 0.037, reg: 1.119, iou: 0.699, data: 1.7e-05, fwd: 1.7e-01, bwd: 9.1e-02, optim: 1.5e-01, : 100%|████████████████████████████████| 2343/2343 [34:56<00:00,  1.12it/s]
2020-07-19 20:29:51.807 | INFO     | videoanalyst.engine.trainer.trainer_base:save_snapshot:143 - Snapshot saved at: /home/ubuntu/Vincent/object_track/video_analyst-master/snapshots/siamfcpp_googlenet-lasot/epoch-0.pkl
  0%|                                                                                                                                                                              | 0/2343 [00:00<?, ?it/s]Traceback (most recent call last):
  File "./main/train.py", line 110, in <module>
    trainer.train()
  File "/home/ubuntu/Vincent/object_track/video_analyst-master/videoanalyst/engine/trainer/trainer_impl/regular_trainer.py", line 97, in train
    training_data = next(self._dataloader)
  File "/home/ubuntu/anaconda3/envs/vincent/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/ubuntu/anaconda3/envs/vincent/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 831, in _next_data
    raise StopIteration
StopIteration
  0%|                                                                                                                                                                              | 0/2343 [00:00<?, ?it/s]
(vincent) ubuntu@ubun:~/Vincent/object_track/video_analyst-master$ python ./main/train.py --config 'experiments/siamfcpp/train/lasot/siamfcpp_googlenet-trn.yaml'

Test the model I have trained

I found that the code base did not declare the code to test my own training model?

SAT training problem

Hi，I'm sorry to bother you again。When i train SAT, and i set num_processes=4, num_workers=16 in sat_res50_davis17.yaml。Unfortunately，it was RuntimeError:
Traceback (most recent call last):
File "./main/dist_train_sat.py", line 211, in
join=True)
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 2 terminated with the following error:
Traceback (most recent call last):
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/yanyi/PaperProject_chen/video_analyst-master/main/dist_train_sat.py", line 121, in run_dist_training
dist_utils.synchronize()
File "/home/yanyi/PaperProject_chen/video_analyst-master/videoanalyst/utils/dist_utils.py", line 81, in synchronize
dist.barrier()
File "/home/yanyi/miniconda3/envs/touch-env/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 1489, in barrier
work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled cuda error, NCCL version 2.4.8

And i set the num_processes=1,it start trianing, but out of CUDA memory.