The course project is to apply object detection in Urban Traffic (e.g. detection of cars)
- Tools: PyTorch, mmDetection tool (from OpenMMLab project)
- Data:
- Pretrained with COCO Dataset: Common Objects in Context
- Fine-tuning in Cityscape dataset
- Implementation details:
- Deformable DETR: Deformable Transformers for End-to-End Object Detection method
- Apply pretrained weights of COCO dataset
Install MMdetection
Follow https://mmdetection.readthedocs.io/en/stable/get_started.html to install the library.
nvidia-smi # check cuda version
conda create --name openmmlab python=3.8 -y
conda activate openmmlab
conda install pytorch torchvision -c pytorch -y
pip install jupyter
pip install -U openmim
mim install mmcv-full
pip install mmdet
- Download Cityscapes Datasets from link
- Download a small test dataset. gtFine_trainvaltest.zip (241MB, ground truth), leftImg8bit_trainvaltest.zip (11GB, raw images)
- put the data with a specific structure folder according to link
repo ├── tools ├── configs ├── data │ ├── cityscapes │ │ ├── annotations │ │ ├── leftImg8bit │ │ │ ├── train │ │ │ ├── val │ │ ├── gtFine │ │ │ ├── train │ │ │ ├── val
- MMdet provides a script to convert Citiscapes to COCO annotation format.
It generates the ground truth with COCO ground truth format into the folder
pip install cityscapesscripts python tools/dataset_converters/cityscapes.py ./data/cityscapes --nproc 8 --out-dir ./data/cityscapes/annotations'
data/cityscapes/annotations
We use the deformable-detr method. First download the pretrained weight using deformable_detr on COCO2017.
mim download mmdet --config deformable_detr_r50_16x2_50e_coco --dest .
It should get the config file deformable_detr_r50_16x2_50e_coco.py and pretrained checkpoint deformable_detr_r50_16x2_50e_coco_20210419_220030-a12b9512.pth.
The new modifed config file is configs/deformable_detr/deformable_detr_cityscapes.py which sets the dataset folder path, changes the images scales accroding to cityscapes, and loads checkpoint from the specific pretrained pth file.
python tools/train.py \
configs/deformable_detr/deformable_detr_cityscapes.py \
--work-dir logs_ckpt/self-de-detr/
With the tensorboard hook on, the training loss/metric figure can be easily checked. It takes around 2 hour to train in a single A100 gpu for one epoch.
Here also tried another lighter method with cityscapes config file which is directly provided by mmdetection. Download the pretrained weight and its config by mim download mmdet --config faster_rcnn_r50_fpn_1x_coco --dest .
python tools/train.py \
configs/faster_rcnn/faster_rcnn_r50_fpn_1x_cityscapes.py \
--work-dir logs_ckpt/faster_rcnn/
bbox_mAP | bbox_mAP_50 | bbox_mAP_75 | bbox_mAP_s | bbox_mAP_m | bbox_mAP_l |
---|---|---|---|---|---|
0.384 | 0.642 | -1.000 | 0.166 | 0.379 | 0.600 |
Since cityscapes has a "rider" class which is quite interesting for us. (Since coco doesn't have rider.) We picked some pictures with GT label "rider", and compare it with before and after finetuned's model in test.py
. The trained pth files can be downloded here
We also try to generate a demo from a video (a sequnce of images) provided by Cityscapes Download leftImg8bit_demoVideo.zip. Refer to test_generate_images_demo.py
.