tostenzel / mcmot-transformer

Multi-camera Multi-object Tracking with Transformer

McFormer: Multi-Camera Multi-object Tracking with Transformers

This repository hosts the implementation of the master's thesis "Multi-Camera Multi-object Tracking with Transformers" written by Tobias Stenzel and supervised by Paul Swoboda.

The codebase builds upon DETR, Deformable DETR, Tracktor, and Trackformer.

Branches

• The default branch 24-switch-mcmot-to-detr containts the final MCMOT model
• Branch 1-train-trackformer-on-wildtrack contains the single-camera model and the results on WILDTRACK
• Branch 22-xminw-yminh-ww-hh-format-with-normal-detr contains a sanity check for the MCMOT because it substitutes the transformation from 2D to 3D and back by predicting bounding box form (xmin/W, ymin/H, w/W, h/H) instead (xcenter/W, ycenter/H, w/W, h/H). It also contains the feature to set the learning rates for each transformer layer separately.

Text

The thesis is accessible only for invitees in this Overleaf project.

Installation

The INSTALL.md describes the installation process on the server with the address dws-student-01.informatik.uni-mannheim.de.

Training

The TRAIN_BASELINE.md describes how to prepare and execute the training of the baseline model on WILDTRACK.

mcmot-transformer's Issues

Train Trackformer on WILDTRACK

• Generate train/test/val split (75%/12.5%/12.5%) for WILDTRACK in COCO and MOT format and check that these are correct
• Register WILDTRACK as dataset in the ML pipeline and check that it works properly (No MOTA below 0)
• Register fine-tuning datasets with additional training data in ML pipeline
• Provide all config.yamls for different training setups for the baseline ablation in the thesis

Another try with 3d training

Working version without preprocessing data and targets

expand training on xywh to training on cylinders

train on 3D only and monitor visually

Create multi-camera 3D training dataset

Feed multiple image sequences at once into trackformer

This requires the following features (different sequences refer to different cameras/views):

• A torch.DataLoader for the whole dataset that can call images from all sequences and not only from one sequence
• The possibility to load images into the CNN backbone by alternating over images of the the same time period from different sequences (possibly in parallel) instead of sequence-by-sequence. For one time period, this will return k token sequences
• The possibility to append the token sequences for one time period and feed it into the transformer

Points 2 and 3 may require to (temporarily) reduce the dimension of the CNN backbone output / transformer input tokens to make this computationally feasible

• Adding the sinusoidal location embedding to each token sequence before appending
• Adding one learned embedding vector to every token. But the k learned embeddings differ per sequence

Train on internal (xmin/W, ymin/H, w/W, h/H) bbox format and make evaluation work

switch MCMOT to DETR

(xmin/W, ymin/H, w/W, h/H) format with normal DETR

eval on multicam 3D and monitor visually

train and eval on 3D multicam WILDTRACK data