angzz / panoptic-fpn-gluon

Panoptic Feature Pyramid Networks

Panoptic Feature Pyramid Networks

This is an unofficial implementation of Panoptic-FPN in a gluon-cv style, we implemented this framework in a fully Gluon API, please stay tuned!

Main Results

Cityscapes

• panoptic_fpn_resnet50_v1b_citys

-	PQ	SQ	RQ	N
All	55.4	77.9	69.3	19
Things	52.4	78.1	66.6	8
Stuff	57.6	77.7	71.2	11

Installation

1. Install cuda 10.0 and mxnet 1.4.0.

sudo pip3 install mxnet-cu100==1.4.0.post0

2. Clone the code, and install gluoncv with setup.py.

cd panoptic-fpn-gluon
sudo python3 setup.py build
sudo python3 setup.py install

Preparation

Cityscapes

1. Download Cityscapes datasets follow the official tutorials and create a soft link.

ln -s $DOWNLOAD_PATH ~/.mxnet/datasets/citys

You can also download from Cityscapes and execute the command above.

2. Create Panoptic images for training and Inference, the code can be found here. Then execute the command below:

python3 createPanopticImgs.py --dataset-folder ~/.mxnet/datasets/citys/gtFine/ --output-folder ~/.mxnet/datasets/citys/gtFine/

Note that the correct data structure is shown below:

$ ls ~/.mxnet/datasets/citys
├── gtFine
│   ├── train/
│   ├── val/
│   ├── test/
│   ├── cityscapes_panoptic_train/
│   ├── cityscapes_panoptic_val/
│   ├── cityscapes_panoptic_test/
│   ├── cityscapes_panoptic_train.json
│   └── cityscapes_panoptic_val.json
├── leftImg8bit
│   ├── train/
│   ├── val/
│   └── test/

3. More preparations can also refer to GluonCV.

4. All experiments are performed on 8 * 2080ti GPU with Python3.5, cuda10.0 and cudnn7.5.0.

COCO

• TODO

Structure

* Model : $ROOT/gluoncv/model_zoo/panoptic/
* Train & valid scripts : $ROOT/scripts/panoptic/
* Metric : $ROOT/gluoncv/utils/metric/

Training & Inference

Cityscapes

1. Clone the training scripts here, then train panoptic_fpn_resnet50_v1b_citys with:

python3 train_panoptic_fpn.py --network resnet50_v1b --use-fpn --gpus 0,1,2,3,4,5,6,7 --num-workers 32 --log-interval 10 --save-interval 20 --val-interval 10 --epochs 700 --lr-decay-epoch 430,590 --lr-warmup 1600

Note that we follow the training settings described in original paper.

2. Clone the validation scripts here, then validate panoptic_fpn_resnet50_v1b_citys with:

python3 eval_panoptic_fpn.py --network resnet50_v1b --gpus 0,1,2,3,4,5,6,7 --pretrained ./XXX.params

COCO

• TODO

Reference

• Panoptic FPN: Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár.
  "Panoptic Feature Pyramid Networks." CVPR (2019 oral). [paper]

panoptic-fpn-gluon's Issues

mxnet.base.MXNetError: value 0 for Parameter num_args should be greater equal to 1, in operator Concat(name="", num_args="0", dim="0")

Traceback (most recent call last):
File "train_panoptic_fpn.py", line 562, in
train(net, train_data, val_data, eval_metric, ctx, args)
File "train_panoptic_fpn.py", line 440, in train
for i, batch in enumerate(train_data):
File "/export/anaconda3/envs/Panoptic-FPN/lib/python3.5/site-packages/mxnet/gluon/data/dataloader.py", line 443, in next
batch = pickle.loads(ret.get()) if self._dataset is None else ret.get()
File "/export/anaconda3/envs/Panoptic-FPN/lib/python3.5/multiprocessing/pool.py", line 644, in get
raise self._value
mxnet.base.MXNetError: value 0 for Parameter num_args should be greater equal to 1, in operator Concat(name="", num_args="0", dim="0")

when I train panoptic_fpn , the above problem happened, I don't know how to solve it.

problem with 'current_map'

In 'train_panoptic_fpn' L522 there is a condition in which in case of its execution the dict 'result' which further is not used anywhere is filled, and in the opposite case the variable 'current_map' which arrives in function 'save_parans' is declared

Any update results?

Hi, Angzz

Many thanks to your awesome-repo and this gluon reimplementation. Are there some updated results that you can share?

Can you provide pfpn with pure resnet101 results on citys as I failed with PQ=52 ?

How to run this model on my custom dataset?

Hi! Thanks for the repo. How can I run this model on my dataset which has /train/images and /train/masks? Take these 3 pictures as an example. The first one is the image and the second one is the mask and the third one is the overlay.

Inference on trained model

Hi, is there any demo.py script or something related to be able to see the inference results on a new image?