zhen-he / tracking-by-animation Goto Github PK

hello, it support real time or online situations? perhaps, could i get the pretrained models?

hi,thanks for your model. I have a question, can this model detect targets? And before test, need I detect targets first?

hello, how should I understand this word 'Tracker state' ? what does it consist of and what does it stand for?

My pytorch vision is 1.0.1 instead of 0.3.1,and when I run the gen_mnist.py ,it seem occur a wrong that:

(pytorch) yeyu@yeyu-QiTianM415-D004:~/downProject/tracking-by-animation$ python scripts/gen_mnist.py 
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to data/mnist/MNIST/raw/train-images-idx3-ubyte.gz
100%|█████████████████████████████▉| 9904128/9912422 [01:29<00:00, 44937.66it/s]Extracting data/mnist/MNIST/raw/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to data/mnist/MNIST/raw/train-labels-idx1-ubyte.gz
                                                                               Extracting data/mnist/MNIST/raw/train-labels-idx1-ubyte.gz                       
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to data/mnist/MNIST/raw/t10k-images-idx3-ubyte.gz
9920512it [01:40, 44937.66it/s]                                                Extracting data/mnist/MNIST/raw/t10k-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to data/mnist/MNIST/raw/t10k-labels-idx1-ubyte.gz
                                                                               Extracting data/mnist/MNIST/raw/t10k-labels-idx1-ubyte.gz
Processing...
Done!t [00:00, 16220.22it/s]                                                    
train frame number: 1920000.0
test frame number: 80000.00000000007
Running with 4 cores.
Traceback (most recent call last):
  File "scripts/gen_mnist.py", line 170, in <module>
    out_batch = parallel(delayed(process_batch)(states_batch[n], s) for n in range(0, N)) # N * 2 * T * H * W * D
  File "/home/yeyu/anaconda3/envs/pytorch/lib/python3.6/site-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/home/yeyu/anaconda3/envs/pytorch/lib/python3.6/site-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/yeyu/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/home/yeyu/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/yeyu/anaconda3/envs/pytorch/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 567, in __call__
    return self.func(*args, **kwargs)
  File "/home/yeyu/anaconda3/envs/pytorch/lib/python3.6/site-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/home/yeyu/anaconda3/envs/pytorch/lib/python3.6/site-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "scripts/gen_mnist.py", line 95, in process_batch
    h_, w_ = round(h * scale * ratio), round(w * scale / ratio)
TypeError: type Tensor doesn't define __round__ method
9920512it [01:57, 84620.34it/s]
1654784it [00:26, 63032.69it/s] 

So I try to solve it by changing h_, w_ = round(h * scale * ratio), round(w * scale / ratio) into h_, w_ = np.round(h * scale * ratio),np. round(w * scale / ratio).Then it seem work,but I don't know if it will affect the result.

Dear @zhen-he

I got an error - seem like related to the reading of data_config.json in the line 80 of the run.py
Do we suppose to generate the file of data_config.json before the line 80 or we should put into the root?

Traceback (most recent call last):
File "run.py", line 85, in
o.exp_config = utils.load_json('modules/exp_config.json')[o.exp]
File "/home/yang/huckcode/tracking-by-animation-master/modules/utils.py", line 57, in load_json
data = json.load(f)
File "/opt/conda/envs/test1/lib/python3.7/json/init.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/opt/conda/envs/test1/lib/python3.7/json/init.py", line 348, in loads
return _default_decoder.decode(s)
File "/opt/conda/envs/test1/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/conda/envs/test1/lib/python3.7/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 9 column 1 (char 175)

now, which url can download DukeMTMC datasets. i can't find unofficial url.

Hi,
I tried running the code to train and I got this error:

(track) PS C:\Users...\trained_models\tracking-by-animation> python run.py --task sprite
Task: sprite
Experiment: tba
Configuration: ['act']
Model: default
Total batch size: 64, GPU number: 2, GPU batch size: 32
Parameter number: 1.018 M
tensor([[0.0649, 0.0000, 0.0000, 0.0000]], device='cuda:0')
C:\Users...\Miniconda3\envs\track\lib\site-packages\torch\nn\parallel_functions.py:61: UserWarning: Was asked to ga
ther along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
warnings.warn('Was asked to gather along dimension 0, but all '
Traceback (most recent call last):
File "run.py", line 265, in
run_train_epoch(batch_id_start)
File "run.py", line 227, in run_train_epoch
loss = run_batch(batch_id, 'train')
File "run.py", line 189, in run_batch
loss, forward_time = forward(X_seq, **kwargs)
File "run.py", line 165, in forward
loss = net(X_seq, **kwargs)
File "C:\Users...\Miniconda3\envs\track\lib\site-packages\torch\nn\modules\module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "C:\Users...\trained_models\tracking-by-animation\modules\net.py", line 78, in forward
loss = self.loss_calculator(X_r_seq, X_seq, area, **ka)
File "C:\Users...\Miniconda3\envs\track\lib\site-packages\torch\nn\modules\module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "C:\Users...\trained_models\tracking-by-animation\modules\loss_calculator.py", line 36, in forward
losses['tight'] = loss_tight.item()
ValueError: only one element tensors can be converted to Python scalars

Below I pasted my environment information. Do you have any recommendations?
Thank you,
Mikey

Environment information:

Name Version Build Channel
backcall 0.2.0 py_0
blas 1.0 mkl
ca-certificates 2020.7.22 0
certifi 2020.6.20 py37_0
cffi 1.14.2 py37h7a1dbc1_0
colorama 0.4.3 py_0
cudatoolkit 10.1.243 h74a9793_0
cycler 0.10.0 py37_0
decorator 4.4.2 py_0
freetype 2.10.2 hd328e21_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
intel-openmp 2020.2 254
ipykernel 5.3.4 py37h5ca1d4c_0
ipython 7.17.0 py37h5ca1d4c_0
ipython_genutils 0.2.0 py37_0
jedi 0.17.2 py37_0
jpeg 9b hb83a4c4_2
jupyter_client 6.1.6 py_0
jupyter_core 4.6.3 py37_0
kiwisolver 1.2.0 py37h74a9793_0
libpng 1.6.37 h2a8f88b_0
libsodium 1.0.18 h62dcd97_0
libtiff 4.1.0 h56a325e_1
lz4-c 1.9.2 h62dcd97_1
matplotlib 3.3.1 0
matplotlib-base 3.3.1 py37hba9282a_0
mkl 2020.2 256
mkl-service 2.3.0 py37hb782905_0
mkl_fft 1.1.0 py37h45dec08_0
mkl_random 1.1.1 py37h47e9c7a_0
ninja 1.10.0 py37h7ef1ec2_0
numpy 1.19.1 py37h5510c5b_0
numpy-base 1.19.1 py37ha3acd2a_0
olefile 0.46 py37_0
opencv-python 4.4.0.42 pypi_0 pypi
openssl 1.1.1g he774522_1
parso 0.7.0 py_0
pickleshare 0.7.5 py37_1001
pillow 7.2.0 py37hcc1f983_0
pip 20.2.2 py37_0
prompt-toolkit 3.0.5 py_0
pycparser 2.20 py_2
pygments 2.6.1 py_0
pyparsing 2.4.7 py_0
pyqt 5.9.2 py37h6538335_2
python 3.7.7 h81c818b_4
python-dateutil 2.8.1 py_0
pytorch 1.4.0 py3.7_cuda101_cudnn7_0 pytorch
pywin32 227 py37he774522_1
pyzmq 19.0.1 py37ha925a31_1
qt 5.9.7 vc14h73c81de_0
scipy 1.5.2 py37h9439919_0
setuptools 49.6.0 py37_0
sip 4.19.8 py37h6538335_0
six 1.15.0 py_0
sqlite 3.33.0 h2a8f88b_0
tk 8.6.10 he774522_0
torchvision 0.5.0 py37_cu101 pytorch
tornado 6.0.4 py37he774522_1
traitlets 4.3.3 py37_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_3
wcwidth 0.2.5 py_0
wheel 0.35.1 py_0
wincertstore 0.2 py37_0
xz 5.2.5 h62dcd97_0
zeromq 4.3.2 ha925a31_2
zlib 1.2.11 h62dcd97_4
zstd 1.4.5 h04227a9_0

Edit: I get the same error when I run python run.py --task mnist instead of python run.py --task sprite

How would I be able to do this? What pre-processing steps should I take?
Thanks

I use camera1 data of Duke_MTMCT to train and test, but result train loss is 105, test loss is 4500. Is there any error with my operation？ or, i need run all cameras to get true result.

When I run python run.py(the --task parameter is mnist or sprite), I'll get RuntimeErrors. Here is the full output of python run.py --task sprite.

(torch03py36) ➜  /home/xxx/Source/tracking-by-animation git:(master) ✗ python run.py --task sprite
Task: sprite
Experiment: tba
Configuration:  ['act']
Model: default
Total batch size: 64, GPU number: 1, GPU batch size: 64
Parameter number: 1.018 M
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1523244252089/work/torch/lib/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
  File "run.py", line 262, in <module>
    run_train_epoch(batch_id_start)
  File "run.py", line 224, in run_train_epoch
    loss = run_batch(batch_id, 'train')
  File "run.py", line 190, in run_batch
    loss, forward_time = forward(X_seq, **kwargs)
  File "run.py", line 167, in forward
    loss = net(X_seq, **kwargs)
  File "/home/linkinpark213/anaconda3/envs/torch03py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/linkinpark213/Source/tracking-by-animation/modules/net.py", line 55, in forward
    C_o_seq = self.feature_extractor(X_seq_cat) # N * T * M * R
  File "/home/linkinpark213/anaconda3/envs/torch03py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/linkinpark213/anaconda3/envs/torch03py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 71, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/linkinpark213/anaconda3/envs/torch03py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/linkinpark213/Source/tracking-by-animation/modules/feature_extractor.py", line 21, in forward
    C3_seq = self.cnn(X_seq)                                             # NT * C3_3 * C3_1 * C3_2
  File "/home/linkinpark213/anaconda3/envs/torch03py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/linkinpark213/Source/tracking-by-animation/modules/submodules.py", line 182, in forward
    H = getattr(self, 'conv'+str(i))(H)
  File "/home/linkinpark213/anaconda3/envs/torch03py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/linkinpark213/anaconda3/envs/torch03py36/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 282, in forward
    self.padding, self.dilation, self.groups)
  File "/home/linkinpark213/anaconda3/envs/torch03py36/lib/python3.6/site-packages/torch/nn/functional.py", line 90, in conv2d
    return f(input, weight, bias)
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1523244252089/work/torch/lib/THC/generic/THCStorage.cu:58

And my environment is:

• Intel(R) Core(TM) i7-4790
• NVIDIA GeForce GTX 1080Ti
• 16GB RAM
• Ubuntu 18.04 (Kernel: 4.15.0-48-generic)
• CUDA 8.0 / 10.0
• Python 3.6.8
• PyTorch 0.3.1

Is it insufficient for running the training code of TBA? Thanks in advance.

Hello Zhen

First of all, I really liked your work and thanks for sharing.

I am trying to train the net on another synthetic dataset where I have moving 3D objects with different sizes on the scene (instead of 2D objects in Sprite). And, I realized that you need to provide an object patch size (Y_{t,i}^a denoted in the paper).

So my question is how can I set this patch size so that it can track objects with different sizes?

i have a question, when a person cross another person in you project, the persons's Id will change, why is it?

Hello, I have run the module and it has remarkable performance. But I am confused about the incentive for the loss to decrease. If the "Feature Extractor" and the "Tracker Array" was trained to a set of parameter in which they always perform Identical output representing any part of the input image, the reconstructed frame rendered according to these output apperance and pose can also be practically equal to the original frame.

Is there any mechanism in the module or in the RAT that motivates the "Feature Extractor" and the "Tracker Array" to output the exact tracking object since it's totally "label-free"?

Many Thanks and Best Regards

I want to run gen_duke.py , but i don't know how to move dukeMTMC to path 'data/duke/processed'.
dukeMTMC
|——detections
|——videos
|——ground_truth
...
these dirs how to replace project data dirs , thanks very much
project/data/duke/processed
|——input_roi
|——input
|——bb
|——bg

When I run python scripts/gen_mnist.py:

Directory data/mnist/pt/input is removed.
Traceback (most recent call last):
  File "scripts/gen_mnist.py", line 53, in <module>
    train_data = torch.load(path.join(input_dir, 'training.pt')) # 60000 * 28 * 28

FileNotFoundError: [Errno 2] No such file or directory: 'data/mnist/processed/training.pt'

I ran python run.py --task mnist and met with this issue. Here is the full output.

(torch03py36) ➜  /home/xxx/Source/tracking-by-animation git:(master) ✗ python run.py --task mnist
Task: mnist
Experiment: tba
Configuration:  ['act']
Model: default
Total batch size: 64, GPU number: 2, GPU batch size: 32
Traceback (most recent call last):
  File "run.py", line 127, in <module>
    net = Net(o).cuda()
  File "/home/linkinpark213/Source/tracking-by-animation/modules/net.py", line 28, in __init__
    self.loss_calculator = nn.DataParallel(LossCalculator(o), device_ids)
  File "/home/linkinpark213/Source/tracking-by-animation/modules/loss_calculator.py", line 17, in __init__
    self.mmd = smd.MMDLoss(sigmas=[0.01, 0.03, 0.1, 0.3, 1, 3])
AttributeError: module 'modules.submodules' has no attribute 'MMDLoss'

By the way, I can't seem to find any MMDLoss in the project, either. Is there any file missing?

Hi,
First, thanks for your great work.
Since I do not have the duke_MTMC data set, I try to build the my data set (50 minutes * 2) with surveillance view as similar as duke_MTMC, and I generate the background by removing the objects detected by the object detection network.
However, my validation curve do not drop, and the gradient of C_o_seq seems vanishing (1.2e-43) at 92890 iteration.
Can you provide some training details? For example, how many GPU do you use? The change of grad_C_o_seq during different iteration? The train time you spend?
I work hard without duke_MTMC, and try to solve the problem :)
Thanks again!