yu-lab-vt / cinda

Circulation Network based Data-association for Multi-object Tracking

Welcome to CINDA

CINDA (CIrculation Network based Data-Association) is a minimum-cost circulation framework for solving the global data association problem, which plays a key role in the tracking-by-detection paradigm of multi-object tracking (MOT). CINDA maintains the same optimal solution as the previously widely used minimum-cost flow framework, while enjoys both a better theoretical complexity bound and orders of practical efficiency improvement. The improved computational efficiency is expected to enable more sophisticated tracking framework and yields better tracking accuracy.

If you have any feedback or issue, you are welcome to either post issue in Issues section or send email to [email protected] (Guoqiang Yu at Virginia Tech).

Overview of CINDA and its comparison with minmum-cost flow-based framework

(a) Objects detected in three consecutive frames. The first frame contains two detections; one missed detection is colored in gray. Lines between detections are the possible ways of linking them. Each line is associated with a cost. If the similarity between two detections is too low to be the same object, we do not link them. There are three trajectories in these three frames. For example, detections 1, 3, and 6 should be linked together as a single trajectory. (b) The proposed minimum-cost circulation formulation for the MOT problem. Detection $x_i$ is represented by a pair of nodes: a pre-node $o_i$ and a post-node $h_i$. The dummy node $s$ is linked to all pre-nodes. Then all post-nodes are linked back to $s$. These edges are shown in dashed lines. Transition edges between detections are shown in blue. Similar to (a), there is no transition edge if the similarity between detections is too low. The input and output flows of every node in the circulation network are balanced. Therefore, the excess for the dummy node is always 0. % and we do not need to specify the expected amount of flow in the network, which is the expected number of tracking targets. (c) Typical minimum-cost flow formulation for MOT problem. Each detection is also represented by a pre-node and post-node. The difference is that in this flow network, there are two dummy nodes: the dummy source node $s$ is linked to all pre-nodes ${o_i}$ and the dummy sink node $t$ is linked to the post-nodes ${h_i}$. The color-coding is the same as that in (b). The input and output flows of these two dummy nodes are both imbalanced and the amount of imbalances decide how many targets we want to track (the amount of flow that can happen in the network). To apply existing minimum-cost flow solvers, we need to specify the excess and deficit of $s$ and $t$ first, which is commonly unknown. (d) The results from the proposed minimum-cost circulation framework. Three trajectories are created and they are shown with the same color as in (a). The solution can be obtained by any minimum-cost circulation solver. (e) The results from the minimum-cost flow formulation. The same three trajectories are generated. In addition to a minimum-cost flow solver, an accompanying searching scheme is needed to find the optimal trajectory number, or equivalently, the optimal flow amount.

Supports to Python and MATLAB

CINDA was implemented using C based on the efficient implementation of cost-scaling algorithm[1]. Interfaces for Python and MATLAB are also provided respectively, on which the efficiency is also guaranteed. Try CINDA now!

Any problem? CINDA does not work on your data? Please open an issue. We are happy to help!

Efficiency comparison

Theoretical bounds compared with minimum-cost flow-based methods

Pracitical efficiency comparisons with minimum-cost flow-based methods

Experiments on natural image benchmarks

Experiments on microscopy imaging data for particle and cell tracking

Case studies using CINDA

1. CINDA makes it possible to do identity inference using more history frames, which retrieve identities of occluded objects
2. CINDA enable us to iteratively refine tracking results on larger scale data (see Table 4)

Cite

Please cite our paper if you find the code useful for your research.

Congchao Wang, Yizhi Wang, Guoqiang Yu, Efficient Global Multi-object Tracking Under Minimum-cost Circulation Framework, arXiv:1911.00796. (accepted by IEEE Trans. on PAMI)

@article{cinda_mot,
  title={Efficient Global Multi-object Tracking Under Minimum-cost Circulation Framework},
  author={Wang, Congchao and Wang, Yizhi and Yu, Guoqiang},
  journal={arXiv preprint arXiv:1911.00796},
  year={2019}
}

Reference

[1].Goldberg, A. V. (1997). An efficient implementation of a scaling minimum-cost flow algorithm. Journal of algorithms, 22(1), 1-29.

[8] S. Sharma, J. A. Ansari, J. K. Murthy, and K. M. Krishna, “Beyond pixels: Leveraging geometry and shape cues for online multiobject tracking,” in 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 3508–3515.

[12] J. Berclaz, F. Fleuret, E. Turetken, and P. Fua, “Multiple object tracking using k-shortest paths optimization,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 9, pp. 1806– 1819, 2011.

[13] L. Zhang, Y. Li, and R. Nevatia, “Global data association for multiobject tracking using network flows,” in 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2008, pp. 1–8.

[14] P. Lenz, A. Geiger, and R. Urtasun, “Followme: Efficient online min-cost flow tracking with bounded memory and computation,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 4364–4372.

[15] H. Pirsiavash, D. Ramanan, and C. C. Fowlkes, “Globally-optimal greedy algorithms for tracking a variable number of objects,” in CVPR 2011. IEEE, 2011, pp. 1201–1208.

[16] A. V. Goldberg, S. Hed, H. Kaplan, and R. E. Tarjan, “Minimum-cost flows in unit-capacity networks,” Theory of Computing Systems, vol. 61, no. 4, pp. 987–1010, 2017.

Updates

1. Sep.2020: Add the support to problems where detections can be shared across different trajectories.

2. Nov.2020: Add support to python 64bit on Windows system (compiled by Mingw-w64 in Cygwin64).

cinda's Issues

About 'res.dF' and 'res.dffMatFav'

lib_cinda_funcs.so error

WhenI try to run the notebook, I get the following error

lib_cinda_funcs.so is either not designed to run on Windows or it contains an error. Try installing the program again using the original installation media or contact your system administrator or the software vendor for support. Error status 0xc000012f

Memory leak in pyCS2() return pointer

The returned pointer from pyCS2() function is never freed. I solved this problem by adding the following function to cinda_funcs.c :

void pyFreeTrackVec(price_t *track_vec) {
free(track_vec);
}

And then, in algo.py, at the end of mcc4mot():

_cinda.pyFreeTrackVec.argtypes = (ctypes.POINTER(ctypes.c_longlong), )
_cinda.pyFreeTrackVec.restype = None
_cinda.pyFreeTrackVec(track_vec)

return traj, cost

how create detecton_mat.txt and transition_mat.txt?

It's a great project, but I have a problem running this project with my own data. If I have a data sequence and the detection results of the sequence, how can I create my own transition_ mat.txt and detecton_mat.txt.

License

Awesome project! I've been doing research myself into same problem.

Would you be able to add a license to the code?

traj, cost = mcc4mot (detection _ arcs, transition _ arcs)

Hello, what an impressive work, thank you for sharing this code!
in the process of calculating traj with traj, cost = mcc4mot (detection _ arcs, transition _ arcs), why are there some cases in which node ids do not have corresponding trajectories in the results of traj?

How to compute C_i (detection_mat)?

C_{i,j}, C_{en}, C_{ex} are well discussed in the paper (how to compute it) but I do not know how you can compute C_i

In your paper, C_i = log(beta_i / (1-beta_i). From your example code, C_i is in range [-8, -6] => beta_i is in range [0.00025, 0.0025]. This means the probability beta_i for detection x_i is approaching to zero.

How can I set beta_i or C_i for each detection in my dataset?

Is it possible to run CINDA with unique IDs but NOT in the range from 1 to n?

Thanks for publishing this offline tracking algorithm.

I think unique IDs in detection_arcs are enough to link detections as a trajectory. Do you have any suggestions to implement CINDA with unique IDs but NOT in the increasing order from 1 to n?
For example, I can have unique IDs in the following order 1, 200000000000, 3, 9, 5 and know how to put them in transition_mat. How can I run CINDA with this input?

1,3.467864,4.1917836,-6.7232538
200000000000,4.2795971,2.9235461,-7.1020659
3,1.7165976,2.4109024,-5.2415588
9,3.6710802,2.5107691,-5.7177933
5,3.3307807,3.359227,-6.3914796

I have tried to run CINDA with this input. It outputted an error (Error 5, wrong value of nodes). I do not know if I ignore (comment) this line (if ( tail < 0 || tail > n || head < 0 || head > n )) can possibly affect the output of CINDA.