jonathonluiten / trackeval

HOTA (and other) evaluation metrics for Multi-Object Tracking (MOT).

TrackEval

Code for evaluating object tracking.

This codebase provides code for a number of different tracking evaluation metrics (including the HOTA metrics), as well as supporting running all of these metrics on a number of different tracking benchmarks. Plus plotting of results and other things one may want to do for tracking evaluation.

NEW: RobMOTS Challenge 2021

Call for submission to our RobMOTS Challenge (Robust Multi-Object Tracking and Segmentation) held in conjunction with our RVSU CVPR'21 Workshop. Robust tracking evaluation against 8 tracking benchmarks. Challenge submission deadline June 15th. Also check out our workshop call for papers.

Official Evaluation Code

The following benchmarks use TrackEval as their official evaluation code, check out the links to see TrackEval in action:

• RobMOTS (Official Readme)
• KITTI Tracking
• KITTI MOTS
• MOTChallenge (Official Readme)
• Open World Tracking (Official Readme)
• PersonPath22

If you run a tracking benchmark and want to use TrackEval as your official evaluation code, please contact Jonathon (contact details below).

Currently implemented metrics

The following metrics are currently implemented:

Metric Family	Sub metrics	Paper	Code	Notes
HOTA metrics	HOTA, DetA, AssA, LocA, DetPr, DetRe, AssPr, AssRe	paper	code	Recommended tracking metric
CLEARMOT metrics	MOTA, MOTP, MT, ML, Frag, etc.	paper	code
Identity metrics	IDF1, IDP, IDR	paper	code
VACE metrics	ATA, SFDA	paper	code
Track mAP metrics	Track mAP	paper	code	Requires confidence scores
J & F metrics	J&F, J, F	paper	code	Only for Seg Masks
ID Euclidean	ID Euclidean	paper	code

Currently implemented benchmarks

The following benchmarks are currently implemented:

Benchmark	Sub-benchmarks	Type	Website	Code	Data Format
RobMOTS	Combination of 8 benchmarks	Seg Masks	website	code	format
Open World Tracking	TAO-OW	OpenWorld / Seg Masks	website	code	format
MOTChallenge	MOT15/16/17/20	2D BBox	website	code	format
KITTI Tracking		2D BBox	website	code	format
BDD-100k		2D BBox	website	code	format
TAO		2D BBox	website	code	format
MOTS	KITTI-MOTS, MOTS-Challenge	Seg Mask	website	code and code	format
DAVIS	Unsupervised	Seg Mask	website	code	format
YouTube-VIS		Seg Mask	website	code	format
Head Tracking Challenge		2D BBox	website	code	format
PersonPath22		2D BBox	website	code	format
BURST	{Common, Long-tail, Open-world} Class-guided, {Point, Box, Mask} Exemplar-guided	Seg Mask	website	format

HOTA metrics

This code is also the official reference implementation for the HOTA metrics:

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking. IJCV 2020. Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixe and Bastian Leibe.

HOTA is a novel set of MOT evaluation metrics which enable better understanding of tracking behavior than previous metrics.

For more information check out the following links:

• Short blog post on HOTA - HIGHLY RECOMMENDED READING
• IJCV version of paper (Open Access)
• ArXiv version of paper
• Code

Properties of this codebase

The code is written 100% in python with only numpy and scipy as minimum requirements.

The code is designed to be easily understandable and easily extendable.

The code is also extremely fast, running at more than 10x the speed of the both MOTChallengeEvalKit, and py-motmetrics (see detailed speed comparison below).

The implementation of CLEARMOT and ID metrics aligns perfectly with the MOTChallengeEvalKit.

By default the code prints results to the screen, saves results out as both a summary txt file and a detailed results csv file, and outputs plots of the results. All outputs are by default saved to the 'tracker' folder for each tracker.

Running the code

The code can be run in one of two ways:

• From the terminal via one of the scripts here. See each script for instructions and arguments, hopefully this is self-explanatory.
• Directly by importing this package into your code, see the same scripts above for how.

Quickly evaluate on supported benchmarks

To enable you to use TrackEval for evaluation as quickly and easily as possible, we provide ground-truth data, meta-data and example trackers for all currently supported benchmarks. You can download this here: data.zip (~150mb).

The data for RobMOTS is separate and can be found here: rob_mots_train_data.zip (~750mb).

The data for PersonPath22 is separate and can be found here: person_path_22_data.zip (~3mb).

The easiest way to begin is to extract this zip into the repository root folder such that the file paths look like: TrackEval/data/gt/...

This then corresponds to the default paths in the code. You can now run each of the scripts here without providing any arguments and they will by default evaluate all trackers present in the supplied file structure. To evaluate your own tracking results, simply copy your files as a new tracker folder into the file structure at the same level as the example trackers (MPNTrack, CIWT, track_rcnn, qdtrack, ags, Tracktor++, STEm_Seg), ensuring the same file structure for your trackers as in the example.

Of course, if your ground-truth and tracker files are located somewhere else you can simply use the script arguments to point the code toward your data.

To ensure your tracker outputs data in the correct format, check out our format guides for each of the supported benchmarks here, or check out the example trackers provided.

Evaluate on your own custom benchmark

To evaluate on your own data, you have two options:

• Write custom dataset code (more effort, rarely worth it).
• Convert your current dataset and trackers to the same format of an already implemented benchmark.

To convert formats, check out the format specifications defined here.

By default, we would recommend the MOTChallenge format, although any implemented format should work. Note that for many cases you will want to use the argument --DO_PREPROC False unless you want to run preprocessing to remove distractor objects.

Requirements

Code tested on Python 3.7.

• Minimum requirements: numpy, scipy
• For plotting: matplotlib
• For segmentation datasets (KITTI MOTS, MOTS-Challenge, DAVIS, YouTube-VIS): pycocotools
• For DAVIS dataset: Pillow
• For J & F metric: opencv_python, scikit_image
• For simples test-cases for metrics: pytest

use pip3 -r install requirements.txt to install all possible requirements.

use pip3 -r install minimum_requirments.txt to only install the minimum if you don't need the extra functionality as listed above.

Timing analysis

Evaluating CLEAR + ID metrics on Lift_T tracker on MOT17-train (seconds) on a i7-9700K CPU with 8 physical cores (median of 3 runs):

Num Cores	TrackEval	MOTChallenge	Speedup vs MOTChallenge	py-motmetrics	Speedup vs py-motmetrics
1	9.64	66.23	6.87x	99.65	10.34x
4	3.01	29.42	9.77x		33.11x*
8	1.62	29.51	18.22x		61.51x*

*using a different number of cores as py-motmetrics doesn't allow multiprocessing.

python scripts/run_mot_challenge.py --BENCHMARK MOT17 --TRACKERS_TO_EVAL Lif_T --METRICS CLEAR Identity --USE_PARALLEL False --NUM_PARALLEL_CORES 1  

Evaluating CLEAR + ID metrics on LPC_MOT tracker on MOT20-train (seconds) on a i7-9700K CPU with 8 physical cores (median of 3 runs):

Num Cores	TrackEval	MOTChallenge	Speedup vs MOTChallenge	py-motmetrics	Speedup vs py-motmetrics
1	18.63	105.3	5.65x	175.17	9.40x

python scripts/run_mot_challenge.py --BENCHMARK MOT20 --TRACKERS_TO_EVAL LPC_MOT --METRICS CLEAR Identity --USE_PARALLEL False --NUM_PARALLEL_CORES 1

License

TrackEval is released under the MIT License.

Contact

If you encounter any problems with the code, please contact Jonathon Luiten ([email protected]). If anything is unclear, or hard to use, please leave a comment either via email or as an issue and I would love to help.

Dedication

This codebase was built for you, in order to make your life easier! For anyone doing research on tracking or using trackers, please don't hesitate to reach out with any comments or suggestions on how things could be improved.

Contributing

We welcome contributions of new metrics and new supported benchmarks. Also any other new features or code improvements. Send a PR, an email, or open an issue detailing what you'd like to add/change to begin a conversation.

Citing TrackEval

If you use this code in your research, please use the following BibTeX entry:

@misc{luiten2020trackeval,
  author =       {Jonathon Luiten, Arne Hoffhues},
  title =        {TrackEval},
  howpublished = {\url{https://github.com/JonathonLuiten/TrackEval}},
  year =         {2020}
}

Furthermore, if you use the HOTA metrics, please cite the following paper:

@article{luiten2020IJCV,
  title={HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking},
  author={Luiten, Jonathon and Osep, Aljosa and Dendorfer, Patrick and Torr, Philip and Geiger, Andreas and Leal-Taix{\'e}, Laura and Leibe, Bastian},
  journal={International Journal of Computer Vision},
  pages={1--31},
  year={2020},
  publisher={Springer}
}

If you use any other metrics please also cite the relevant papers, and don't forget to cite each of the benchmarks you evaluate on.

trackeval's Issues

Minor erros in scripts indication

Hi!

Within the context of PR #36, I ran every scripts in scripts folder. I found out some minor errors in comment to run the script from the data folder. What I understood is those lines at the beginning of each script are indications how to run evaluation scripts for each dataset. So I believe it would be better for new users of trackeval if these errors could be fixed.

• run_bdd.py:
  Line 5: run_bdd.py --USE_PARALLEL False --METRICS Hota --TRACKERS_TO_EVAL qdtrack -> run_bdd.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL qdtrack.
  • Hota is mistyped, must be HOTA.
• run_kitty.py:
  Line 5: run_kitti.py --USE_PARALLEL False --METRICS Hota --TRACKERS_TO_EVAL CIWT -> run_kitti.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL CIWT.
  • Hota is mistyped, must be HOTA.
• run_kitty_mots.py:
  Line 5: run_kitti_mots.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL trackrcnn -> run_kitti_mots.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL track_rcnn --GT_FOLDER data/gt/kitti/kitti_mots_train.
  • --GT_FOLDER must match data folder.
  • --TRACKERS_TO_EVAL has a typo.
• run_mot_challenge.py:
  Line 5: run_mot_challenge.py --USE_PARALLEL False --METRICS Hota --TRACKERS_TO_EVAL Lif_T -> run_mot_challenge.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL MPNTrack.
  • --TRACKERS_TO_EVAL Lif_T doesn't match tracker in data folder.
  • Hota is mistyped, must be HOTA.
• run_mots_challenge.py:
  Line 4: run_mots.py --USE_PARALLEL False --METRICS Hota --TRACKERS_TO_EVAL TrackRCNN -> run_mots_challenge.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL track_rcnn.
  • --TRACKERS_TO_EVAL Lif_T doesn't match tracker in data folder.
  • Hota is mistyped, must be HOTA.
  • Python script name is mistyped.

Have a great day!

Mathias.

bug in intersection_over_area calculation

Hi, I found a bug here:

TrackEval/trackeval/datasets/_base_dataset.py

Line 273 in d86fa9e

ioas = np.zeros_like(intersection)

if bboxes1 and bboxes2 both have whole numbers, then ioas will have np.int dtype, and so at the end all values will be zeros (or 1 if a particular box is 100% inside the other box)

To fix this, let's do this:
ioas = np.zeros_like(intersection).astype(np.float)
OR (maybe)
convert bboxes1 and bboxes2 to float at the very beginning.

custom data annotation?

I want to annotate my custom data to make a similar format to the MOT data format.
Are there any open source/free annotation tools?

Thanks

Running eval given two txt files with results

Hello, thanks for your work!

Can I run the evaluation having results stored as txt files? I need something like: HOTA = (results.txt, ground_truth.txt). Thanks in advance!

how to evaluate multi-class multi-object 2D-tracking task on custom dataset

i try to use run_mot_challenge.py tool to evaluate my tracker on my custom dataset.
Before evalutation, i have change default_dataset_config['CLASS_TO_EVAL'], self.valid_classes, self.class_name_to_class_id, distractor_class_name etc in mot_challenge_2d_box.py.
But the output MOTA is negative, and i dont know how to solve it. Can you help me？thks

YouTube-VIS 2019 train_sub_split.json

In the README Quickly evaluate on supported benchmarks , the gt data (~150mb) is provided such as YouTube-2019

data/gt/youtube_vis/youtube_vis_train_sub_split/train_sub_split.json.

When I look into this gt file, I find the length of videos is 302, same as the YouTube-VIS 2019 validation set provided in CodaLab. And, after a detailed checking, I find they are two different video ID sets. So, Does that mean the data you provied is just part of training set (sample 302 videos from training 2238 videos) ? @JonathonLuiten

Also, what about the STEm_Seg tracker results?

data/trackers/youtube_vis/youtube_vis_train_sub_split/STEm_Seg/data/results.json

Does that mean the above prediction results just are inferenced by the default model provied in STEm_Seg repo or a new model trained by remaining set (2238-302=1936 videos) ?

Thanks for your contribution for Tracking research.

Modify objective function for matchings

Hello,

I have a question regarding the optimisation of HOTA for matchings with the Hungarian algorithm.
Aside from MOT performance, I would like to use TrackEval to obtain raw framewise detection scores (DetRe and DetPr). I do read that matching is built "such that as a first objective the number of TPs is maximised" but I would also like to remove the influence of the association mean to obtain a true "track independant" detection-score before running TrackEval the normal way.

Could you point me to the code modifications I can make for that purpose ? In hota.py, I found the following line but I am not sure what to change:
score_mat = global_alignment_score[gt_ids_t[:, np.newaxis], tracker_ids_t[np.newaxis, :]] * similarity

Thank you very much iin advance !

Mathis

HOTA for Multi Camera

Hi, Thanks for the code, it works like a charm for mono-camera.

I would like to compute HOTA for multi camera dataset (let's say we have 2 seqs corresponding to camera 1 and camera 2 sequence). What should I do?

I had some ideas like :

• each camera sequence as an independant sequence (like 2 mono camera sequence)
• do one sequence which is the concatenation of both sequence ?
• Simultaneously sequence 1 and sequence 2 on the same .txt. Like 1 ID appears on two frames.

I think option 3 would definetely messus up with HOTA computation.
I think that option 1 and 2 would work but would it take in consideration that one ID is the same per frame accross all cameras. I mean if I have an ID_1 on frame 1 on both camera then it takes in consideration it is ID_1 on both camera and not ID_1 and ID_2 for exemple.
We could have a situation like : ID_1 and ID_2 is perfectly tracked in each camera sequence but my tracking advice would still label two differents IDs for the same ID across camera ...

That would be to use HOTA for Person-Reidentification task as Market1501 dataset and so on. In the article it says that HOTA can be easily extended for multi camera, I am curious about this extension and I am sorry if the answer is trivial.

I hope this is clearn and thanks again for your amazing work.

MOTS evaluation script

Hey there,

I've been using your repo for quite a while for MOT challenge. However, I'm facing problems with MOTS challenge.

I've downloaded the data folder to run some simple examples.

When I run (which is a slight fix of what you've mentioned in the comments in run_mots_challenge.py)
python scripts/run_mots_challenge.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL track_rcnn

I'm getting the following error:

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Evaluate on custom video

Hi
I want to use one of the MOTChallenge videos randomly e.g. MOT17-11-SDP to test my tracker's performance. ( i use videos that are for training, so I can get the evaluation metrics like HOTA since the ground truth files are available). I test some trackers like ByteTrack on custom data and want to use TrackEval for evaluation. I'm sure that the output file of the ByteTrack is compatible with what TrackEval requires.
As I use their pretrained models and just test the model on MOTChallenge training videos, I expect to get results almost the same as what they said in their paper (MOTA = 80, IDF1 = 77.3, and HOTA=63.1). but after evaluation using this repository, I get strange results as shown in the following image.
I searched a lot for this problem but couldn't find any helpful advice. is there any step or preprocessing which I forgot to handle?? or the problem is related to the TrackEval??
thanks

3D multi-object tracking

Hi,

Thanks for the amazing repo. It helps a lot. I have been using the code here for 2D box tracking evaluation in KITTI format dataset. However, it seems there is no implmentation for 3D box tracking in pointcloud (Although such data was given in KITTI). Did I miss something?

What if there is comma in the name of sequence?

Hello,

thank you for your great work and contribution to MOT community.

As stated in the title, evaluator parses the name of the sequences by using csv library. Is there a valid assumption that there is no comma in the name of the sequence? If it is stated somewhere in the README or project, I am sorry for the hassle.

Thanks,

help to evaluate on custom dataset

I am using the structure mentioned in comment number #29 with following command to run

!python TrackEval/scripts/run_mot_challenge.py --DO_PREPROC False --BENCHMARK cam1 --SPLIT_TO_EVAL all --TRACKERS_TO_EVAL bytetrack

this is the error i get :

trackeval.utils.TrackEvalException: Ground-truth data contains the following invalid timesteps in seq seq-01: 430, , 431, , 432, , 433, , 434, , 435, , 436, , 437, , 438, , 439, , 440, , 441, , 442, , 443, , 444, , 445, , 446, , 447, , 448, , 449, , 450, , 451, , 452, , 453, , 454, , 455, , 456, , 457, , 458,

these numbers are the frame ids

my gt file looks like this
372,1,1224.45,414.41,55.549999999999955,71.69999999999999,1,1,1.0
373,1,1208.7,414.41,71.29999999999995,71.69999999999999,1,1,1.0
374,1,1178.3,414.41,101.70000000000005,71.69999999999999,1,1,1.0
375,1,1141.8,414.41,138.20000000000005,71.69999999999999,1,1,1.0
376,1,1102.9,402.26,177.0999999999999,71.69,1,1,1.0
377,1,1085.28,400.44,177.10000000000014,71.69,1,1,1.0
378,1,1067.66,398.61,177.0999999999999,71.69999999999999,1,1,1.0
379,1,1034.85,392.54,177.10000000000014,71.69,1,1,1.0
380,1,1003.25,377.95,177.0999999999999,71.69999999999999,1,1,1.0

my tracker out put looks like this
372,1,1002,138,25,23,1,1,1
372,2,367,219,87,41,1,1,1
372,3,683,283,25,24,1,1,1
372,4,1196,419,83,58,1,1,1
372,5,261,57,53,25,1,1,1
372,6,368,219,88,42,1,1,1
372,7,808,177,44,24,1,1,1
372,8,598,206,60,27,1,1,1
372,9,882,170,39,22,1,1,1

any help is appreciated :)

Run from scripts folder does not work

I created a clean virtual environment with python3, installed numpy and scipy. I also downloaded the data.zip and placed it in the main directory of this repository (same level as docs, scripts, tests, trackeval).
When I run the script from the main folder this happens:

./venv/bin/python3 scripts/run_kitti.py

Eval Config:
USE_PARALLEL : False
NUM_PARALLEL_CORES : 8
BREAK_ON_ERROR : True
RETURN_ON_ERROR : False
LOG_ON_ERROR : /home/ias/git/TrackEval/error_log.txt
PRINT_RESULTS : True
PRINT_ONLY_COMBINED : False
PRINT_CONFIG : True
TIME_PROGRESS : True
DISPLAY_LESS_PROGRESS : True
OUTPUT_SUMMARY : True
OUTPUT_EMPTY_CLASSES : True
OUTPUT_DETAILED : True
PLOT_CURVES : True

Kitti2DBox Config:
PRINT_CONFIG : True
GT_FOLDER : /home/ias/git/TrackEval/data/gt/kitti/kitti_2d_box_train
TRACKERS_FOLDER : /home/ias/git/TrackEval/data/trackers/kitti/kitti_2d_box_train/
OUTPUT_FOLDER : None
TRACKERS_TO_EVAL : None
CLASSES_TO_EVAL : ['car', 'pedestrian']
SPLIT_TO_EVAL : training
INPUT_AS_ZIP : False
TRACKER_SUB_FOLDER : data
OUTPUT_SUB_FOLDER :
TRACKER_DISPLAY_NAMES : None

Evaluating 1 tracker(s) on 21 sequence(s) for 2 class(es) on Kitti2DBox dataset using the following metrics: HOTA, CLEAR, Identity, Count

Evaluating CIWT

Tracker CIWT was unable to be evaluated.
Ground-truth data contains the following invalid timesteps in seq 0000: 0,
Traceback (most recent call last):
File "/home/ias/git/TrackEval/trackeval/eval.py", line 87, in evaluate
metric_names)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/eval.py", line 187, in eval_sequence
raw_data = dataset.get_raw_seq_data(tracker, seq)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/datasets/_base_dataset.py", line 87, in get_raw_seq_data
raw_gt_data = self._load_raw_file(tracker, seq, is_gt=True)
File "/home/ias/git/TrackEval/trackeval/datasets/kitti_2d_box.py", line 187, in _load_raw_file
[str(x) + ', ' for x in extra_time_keys]))
trackeval.utils.TrackEvalException: Ground-truth data contains the following invalid timesteps in seq 0000: 0,
Traceback (most recent call last):
File "scripts/run_kitti.py", line 86, in
evaluator.evaluate(dataset_list, metrics_list)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/eval.py", line 177, in evaluate
raise err
File "/home/ias/git/TrackEval/trackeval/eval.py", line 87, in evaluate
metric_names)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/eval.py", line 187, in eval_sequence
raw_data = dataset.get_raw_seq_data(tracker, seq)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/datasets/_base_dataset.py", line 87, in get_raw_seq_data
raw_gt_data = self._load_raw_file(tracker, seq, is_gt=True)
File "/home/ias/git/TrackEval/trackeval/datasets/kitti_2d_box.py", line 187, in _load_raw_file
[str(x) + ', ' for x in extra_time_keys]))
trackeval.utils.TrackEvalException: Ground-truth data contains the following invalid timesteps in seq 0000: 0,

error on MOT evaluation using run_mot_challenge.py

Hello ,
Im getting following error with this command
!python scripts/run_mot_challenge.py --DO_PREPROC False --BENCHMARK MOT20-train

raise TrackEvalException('no seqmap found: ' + os.path.basename(seqmap_file))
trackeval.utils.TrackEvalException: no seqmap found: MOT20-train-train.txt

Looking forward to your respond

How can I generate det.txt file(as in MOT dataset) for my own data frames??

Evaluating a point-based method

Hello,

Is there an easy workaround to use this kit for a tracking method which only detects 2D points ?

From what I see, given bounding boxes coordinates it's easy to rewrite the function computing similarity scores to use Euclidean distance between the centers, but other than that it seems that I will have to specify "fake" bounding box width and heights if I want to comply with any of the dataset formats.

Thanks in advance.

Need help evaluating on custom dataset

Hi!

I'm currently trying to use this great repository to evaluate the performance of my own tracker on a dataset i made myself.

The results from the tracker is in this format (which i believe is correct):
A snippet from my results.txt file is shown below.

2,1,0,227,107,129,-1,-1,-1,-1
2,2,2,0,631,359,-1,-1,-1,-1
2,3,367,211,75,28,-1,-1,-1,-1
3,1,0,226,108,130,-1,-1,-1,-1
3,2,4,0,630,358,-1,-1,-1,-1
3,3,365,211,75,28,-1,-1,-1,-1

As far as i understand the ground truth file is supposed to have the same format (which is has).

However, i'm having troubles with running the run_mot_challenge.py script:
raise TrackEvalException('ini file does not exist: ' + seq + '/' + os.path.basename(ini_file))
trackeval.utils.TrackEvalException: ini file does not exist: ShipSeq-01/seqinfo.ini

It seems like it can't locate the seqinfo.ini file even though it is there. It might be because im very confused about the seqmaps and seqinfo files and how i'm supposed to modify them to my own dataset. Do you have any ideas of what i should do?

Regards

Regarding track mAP on TAO dataset

Hi,

Thanks for the excellent work! I found HOTA had become one of the most popular metrics for MOT nowadays, everyone in the tracking community is using it.

I am recently trying to test trackers on the TAO dataset. When I run TrackEval and the official TAO evaluation script, I find their outputs of track mAP are inconsistent. I am not sure if this is due to my mistake, or if the two implementations of track mAP are actually different? Should I expect the two results to match perfectly?

By the way, I used the off-the-shelf results from AOA, and test on the val split. The output of the official eval code is around 29 mAP, while the output of this codebase is around 16 mAP.

How to do “Matching to Optimise HOTA”

It is easy to calculate hota metrics after detection/gt match. But it is quite confusing about matching rule.
From the paper, hota is calculated independently under different iou threshold for 2d bbox tracking. And the Eq15 is to do Hungarian algorithm on Amax add weighted iou.
But it is some kind of Amax multiply iou in this repo.
It is hard to understand why it's guaranteed to meet maximum hota on all possible detection/gt match, both on paper and code.
For a tiny example on iou 0.5 threshold:

data['gt_ids'] = [np.array([0])] * 9 + [np.array([0,1])]

data['tracker_ids'] = [np.array([0])] * 10

data['similarity_scores'] = [np.array([[1]])] * 9 + [np.array([0.49,0.51]).reshape(2,1)]

data['num_tracker_ids'] = 1

data['num_gt_ids'] = 2

In the example, the code will discard last frame detection under iou 0.5 threshold, DetA 0.75 AssA 0.818 HOTA 0.783
But just let last frame detection to match gt id 1: DetA 0.909 AssA 0.746 HOTA 0.8237

Just do linear_sum_assignment(-similarity) to reproduce.

HOTA on MOT Challenge Benchmark

Hi. Thank you for the great work.
Do you have a plan to provide official HOTA results when we submit to the MOT Challenge Benchmark?

How to build a directory structure suitable for kitti mot datasets？

First, thanks for your great job! I have a question for kitti mots datasets, I don't the content of the directory structure, such as the name, content and position of seqmap file.

GT not loaded for ai_city data in MOTChallenge format

Hi,

Thank you for your great work with the HOTA metrics.
I've a related issue but this is not about the difference between 9 and 10 values per line (even though I tried to remove one to see if it helps, but it does not).

I tried your script with a GT file from the AICity challenge 2021 and compared it with a pred file they provide, both following the MOT format (attached to this message), but for some unknown reason, the script does not read the GT file, same as if it was empty:

Count: DeepSORT-SSD512-pedestrian  Dets      GT_Dets   IDs       GT_IDs
s01-c003                           14151     0         219       0
COMBINED                           14151     0         219       0

I tried to change some value, indexed by obj_id instead of frame_id, remove one '-1' at the end of each line, checked the encoding and also checked the entire hierarchy of 'trackers' and 'gt' folders.

Finally, I copied a GT file from a MOT sequence (MOT16-train\MOT16-02) and pasted it as it was my own GT file, even if it doesn't match at all... and it reads the GT file correctly:

Count: DeepSORT-SSD512-pedestrian  Dets      GT_Dets   IDs       GT_IDs
s01-c003                           14151     17833     219       54
COMBINED                           14151     17833     219       54

Here's the original GT file of the sequence I want to test:
gt.txt

Here's one of the prediction file:
s01-c003.txt

Thank you in advance for your help.

Originally posted by @JonathanSamelson in #23 (comment)

GT files format for MOT

In the description file, it's written that the ground truth/ prediction files should contain 10 values per row for MOT sequences (15, 16, 17, 20): link.

However, in the data.zip mentioned in the README, GT files data/gt/mot_challenge/ MOT16-train, MOT17-train, MOT20-train contain only 9 values per row.

Is it an intended thing or can be considered as an issue?

Evaluation on custom dataset KITTI-like

Hi
I'm trying to evaluate tracking results on a custom dataset which has been converted to the KITTI 2d format.
I downloaded the example data for some clarity on the structure the data should have but I can't figure out what the evaluate_tracking.seqmap.test file is representing and how I should configure it for my custom dataset.
Hope anyone can help on this,
thanks in advance

Extremely Low Metric Values

@JonathonLuiten

Hey, hope all is fine. Well I am trying to evaluate my online deepsort tracker. The method uses a separate object detector and thus cannot use premade detections for feature extraction and tracking, but rather live input from a detector were my input video is the MOT16-02 Video. Therefore, some frames are missed and thus removed these frames from the Gt file not to account for missed frames, but the missed detections are not removed from the Gt.

The evaluation output values are so low such that the HOTA is 1.6, MOTA is negative 26.23 and IDF1/IDR/IDP/IDTP values are 0. Is this because I am not using the same pre-generated detections ? And is there another evaluation method that doesnt need any precomputed Detections ?

Why is 'matches count' used twice for AssA calculation

I received the following question via email. I thought I would share the question and my answer here as an issue in case anyone is wondering the same. - Jono

hi, I am reading your paper of hota tracking metric. I have one question about how AssA is calculated.

As the pic below shown, A(c) is ass_a in your code. The paper says AssA equals the sum of all A(c) / TP, but an extra matches_count is timed to every A(c) before sum, which is different from what in paper.

Could you tell me why? thanks.

Filepath in test_all_quick.py may be wrong

In the 48th line of test_all_quick.py, you writetest_data_loc = os.path.join(os.path.dirname(__file__), '..', 'data', 'tests', file_loc), but under the \data directory, there are no \tests but only \trackers, so an FileNotFoundError occurred.

What are seqmap_file and seq_info

Hi, this is a great job!

However, I found some parameters not familiar for me, could you please explain what files they correspond to, like in the MOT dataset, thanks!

They are seqmap_file and seq_info in the code here.

MOT Test Data

Thank you very much for the code and data packages, I was wondering if the Test data for the MOT Benchmarks are available ? And how to adapt the code to run only one sequence of one benchmark of one tracker ?

Can I evaluate MODA metric with this repo?

Hi,
Can I evalutate MODA metric using this code?
MODA (Multi-Object Detection Accuracy): the metric for object detection. (MOT17Det, MOT20Det)

mutil class

how to evaluate mutil class dataset?

MOT 2D data format: world coordinate or class id

The MOT challenge format description for tracking with bounding boxes on the MOT challenge website as well as here: https://github.com/JonathonLuiten/TrackEval/blob/master/docs/MOTChallenge-format.txt say that the columns are:

<frame>, <id>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>, <z>
...
The world coordinates x,y,z are ignored for the 2D challenge and can be filled with -1.

However, putting -1 like the example in the column of ground truth data causes TrackEval to throw an exception. On inspection of the code, it seems that this column is actually being interpreted as object class (lines 249-250 in datasets/mot_challenge_2d_box.py):

               if time_data.shape[1] >= 8:
                    raw_data['classes'][t] = np.atleast_1d(time_data[:, 7]).astype(int)

I wonder if the MOT format changes for ground truth data, or if I am missing something.

Negative values of MOTA, and Zeros of MOTP

Hi,

Thank you for your great work. Its very helpful

I have evaluated People detection and tracking solution based on PeopleNet + KLT (nvtracker) using TrackEval on MOT Challenge in CLEAR metrics for data MOT20 training

Tracker outputs
MOT20-01.txt
MOT20-02.txt
MOT20-03.txt
MOT20-05.txt

Metric results
pedestrian_detailed.csv
pedestrian_summary.txt

As can be seen in the attached results. I don’t understand why i have sometimes ‘-’ or ‘+’ in MOTA There are also 0 values of MOTP, which signals an error in calculations (?)

Segmentation GT masks for MOT-17 OR Tracking GT annotations for MOTS

Hello Dear Authors,
Thank you for your effort in creating the dataset.

I am trying to use MOT17 data for my project and I was wondering if you have segmentation masks for the frames along with tracking/detection annotations?

Also, I see that MOTS has the segmentation masks but not sure if it has tracking and detection annotations?

Could you please clarify on both the fronts?
Thank you in advance.

Problem script run_davis.py

Hi!

Within the context of PR #36 I found the following problem. I found out that I can't execute scripts/run_davis.py, I have the following ouput error:

Eval Config:
USE_PARALLEL         : False                         
NUM_PARALLEL_CORES   : 8                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /home/muffie/code/TrackEval-master/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : True                          
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : False                         

DAVIS Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /home/muffie/code/TrackEval-master/data/gt/davis/davis_unsupervised_val/
TRACKERS_FOLDER      : /home/muffie/code/TrackEval-master/data/trackers/davis/davis_unsupervised_val/
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['ags']                       
SPLIT_TO_EVAL        : val                           
CLASSES_TO_EVAL      : ['general']                   
TRACKER_SUB_FOLDER   : data                          
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FILE          : None                          
SEQ_INFO             : None                          
MAX_DETECTIONS       : 0                             

Evaluating 1 tracker(s) on 30 sequence(s) for 1 class(es) on DAVIS dataset using the following metrics: HOTA, Count


Evaluating ags

Tracker ags was unable to be evaluated.
numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Traceback (most recent call last):
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 86, in evaluate
    res[curr_seq] = eval_sequence(curr_seq, dataset, tracker, class_list, metrics_list,
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 187, in eval_sequence
    raw_data = dataset.get_raw_seq_data(tracker, seq)
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/datasets/_base_dataset.py", line 87, in get_raw_seq_data
    raw_gt_data = self._load_raw_file(tracker, seq, is_gt=True)
  File "/home/muffie/code/TrackEval-master/trackeval/datasets/davis.py", line 123, in _load_raw_file
    from pycocotools import mask as mask_utils
  File "/home/muffie/.local/lib/python3.8/site-packages/pycocotools/mask.py", line 3, in <module>
    import pycocotools._mask as _mask
  File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Traceback (most recent call last):
  File "scripts/run_davis.py", line 90, in <module>
    evaluator.evaluate(dataset_list, metrics_list)
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 177, in evaluate
    raise err
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 86, in evaluate
    res[curr_seq] = eval_sequence(curr_seq, dataset, tracker, class_list, metrics_list,
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 187, in eval_sequence
    raw_data = dataset.get_raw_seq_data(tracker, seq)
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/datasets/_base_dataset.py", line 87, in get_raw_seq_data
    raw_gt_data = self._load_raw_file(tracker, seq, is_gt=True)
  File "/home/muffie/code/TrackEval-master/trackeval/datasets/davis.py", line 123, in _load_raw_file
    from pycocotools import mask as mask_utils
  File "/home/muffie/.local/lib/python3.8/site-packages/pycocotools/mask.py", line 3, in <module>
    import pycocotools._mask as _mask
  File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

---

Steps to reproduce:

• git clone https://github.com/JonathonLuiten/TrackEval
• add data folder to root dir of the repo folder
• install local venv with depencies : numpy==1.18.1, scipy==1.4.1, pycocotools==2.0.2, matplotlib==3.2.1, Pillow==8.1.2 according to requirements.txt
• python scripts/run_davis.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL ags --PLOT_CURVES False

Also, DAVIS evaluation needs pycocotools which is not in minimal_requirements.txt. Also pycocotools module relies on matplotlib + pillow. According to this README, I think these depencies should be added to minimal requirements in order to run on DAVIS dataset as intended

Update: Same problem with KITTI MOTS, MOTS Challenge and Youtube VIS. They both needs pycocotools that seems a core package for segmentation evaluation. And it got the exact same error.

Different MOTA score when using ByteTrack and TrackEval?

Hi, thanks for your amazing work.

But I got different MOTA score when using evaluation code in ByteTrack and TrackEval (72.5 vs. 73.275).
Do you know the reason of that? Or there are some parameters to set?

How do we get graphs such as 'AssA bs DetA' ?

Like Kitti dashboard, How do we get graphs such as 'AssA bs DetA', 'AssPr vs AssRe', 'DetPr vs DetRe', and 'HOTA(0) vs LocA(0)'?
please, let me know a python script.

OHOTA implementation?

The paper mentions OHOTA for online tracking. Is it implemented in TrackEval? If so, is there a flag or command-line option for computing this instead of HOTA?

Did you support the metrics for evaluating MOT17Det, MOT20Det?

Thank you for your great work! I wonder whether your metrics can measure the detection and tracking performance for MOT17Det and MOT20Det? Thank you so much!

ModuleNotFoundError: No module named 'eval_code'

Hi, Thank you for sharing your code, it is an amazing work!
Yet when I run the code, I got the following error:
Traceback (most recent call last): File ".\eval_code\Scripts\run_MOTChallenge.py", line 59, in <module> from eval_code.Datasets.MOTChallenge_Dataset import load_raw_MOTCha_seq, preproc ModuleNotFoundError: No module named 'eval_code'
May I ask the reason?

whether this repo support waymo and nuScenes dataset?

nuScenes:https://www.nuscenes.org/tracking/?externalData=all&mapData=all&modalities=Any
waymo:https://waymo.com/open/challenges/2020/3d-tracking/

_calculate_similarities crashes when tracker_dets_t is empty

For some frames when there are no tracks (tracker_dets_t empty), _calculate_similarities calls either _calculate_box_ious or _calculate_mask_ious assuming both contains objects

conf value and det.txt what are they?

I am trying to understand what format my tracker output should be:

in 1. its confusing for me, conf and what is det.txt and in 2. here conf value is -1.

my question, in my tracker output do i print conf as -1 ? and what does this statement mean? "The conf value contains the detection confidence in the det.txt files." i am aware that my detector can produce txt files with confidence value. so in the tracker output i don't print conf value and make it -1 and let my detector print conf value?

frame, id, bb_left, bb_top, bb_width, bb_height, conf, x, y, z
The conf value contains the detection confidence in the det.txt files.

Tracking with bounding boxes
(MOT15, MOT16, MOT17, MOT20)
1, 3, 794.27, 247.59, 71.245, 174.88, -1, -1, -1, -1
1, 6, 1648.1, 119.61, 66.504, 163.24, -1, -1, -1, -1
1, 8, 875.49, 399.98, 95.303, 233.93, -1, -1, -1, -1
...

About locA score

Hi Jonathon,

I tried use ground truth files as tracker prediction result to test the code. For most metrics, including HOTA, they are 100%, which is reasonable. But for locA score under HOTA_AUC section, it's over 99%, but not 100%.

The ground truth file I use is MOT17-04-DPM, and I remove all other classes, except pedestrain class as the tracker's prediction file. I use earlier version code before current version. Could you explain it, thanks!

Handling occlusions in the Ground Truth

Hi guys,
Currently using this code to evaluate my MOT system and really liking the results!
I have a dataset where a lot of occlusions happen which have been labelled.

I was wondering if there is a system in place to correctly handle that information in the Ground-Truth.
Any advice would be wonderful!
Regards,
Max

SORT Paper Scores

Hello,

I have been experiencing difficulties to reproduce MOTChallenge scores of SORT tracker for MOT15 dataset. I used det.txt provided in git repo of the paper (frcnn detections). I have attached my results in .csv and .txt formats. Any help would be appreciated. Thanks.

pedestrian_detailed.csv
pedestrian_summary.txt

https://arxiv.org/pdf/1602.00763.pdf

Fatih

Bug in identity.py

Hi, I thought I found a bug in TrackEval/trackeval/metrics/identity.py.

Current Version
The fn_mat[:num_gt_ids, :num_tracker_ids] and fp_mat[:num_gt_ids, :num_tracker_ids] entries are not right.

Currently,

fn_mat[:num_gt_ids, :num_tracker_ids] = number_of_gt_ids - potential_matches_count
fp_mat[:num_gt_ids, :num_tracker_ids] = number_of_tracker_ids - potential_matches_count

However, this is not what the original paper explains (click here). This does not handle the case when there are no potential matches.

Two regular nodes are connected with an edge e ∈ E if their trajectories overlap in time.

What I think It Should Be

There should be an if-else statement which checks whether trajectories overlap in time.

# Assume 1e10 is infinity
fn_mat[:num_gt_ids, :num_tracker_ids] = (number_of_gt_ids - potential_matches_count) if (potential_matches_count > 0) else 1e10
fp_mat[:num_gt_ids, :num_tracker_ids] = (number_of_tracker_ids - potential_matches_count) if (potential_matches_count > 0) else 1e10

I have been quite handwavy with the code notation, but hopefully it still conveys the meaning.

Is my understanding right?

Questions about the data

Hey,

First and foremost, thank you for this work to facilitate analysis of multiobject trackers. After having a look at the readme-files, code, and provided data I have a couple questions.
i) The RobMOTS data, train_data.zip, seems to contain detections, ground-truth, and the predictions made by the simplest possible tracker. The detections are missing on BDD100k and waymo. The ground-truth seems to be missing for waymo. Are the missing detections and ground-truth incoming?
ii) Are the corresponding val_data.zip and test_data.zip missing? Will at least the validation set be provided with ground-truth?
iii) I noticed that there are no corresponding images provided. For the RobMOTS challenge, are we expected to rely only on the detections? For tracker evaluation and analysis in general, do you plan to supply also matching images?
iv) On that note, it seems straightforward to download images myself and match them to the provided detections, as the detection files are named after the sequence names. However, how do I match the image frames to the time steps? Specifically, the YouTubeVIS detection times do not seem to correspond to the YouTubeVIS dataset times.
v) Both the TAO-dataset and BDD100k are utilized. Is this not problematic as TAO contains some sequences from BDD100k?

Best regards,
Joakim

MOTS PNG Support

Hi Jonathon,

Great work! I was just wondering if there would be any support for evaluating MOTS when the ground truths are in MOTS PNG format?
I have a custom dataset I would like to evaluate with HOTA, but it is annotated in the MOTS PNG format just like KITTI MOTS, and I don't have the MOTS txt's handy with me because the annotation tool doesn't offer exports to that format. Unless there is a way to convert the MOTS PNG to MOTS txt, I don't know how to evaluate my dataset with HOTA.

Thanks a lot!