angzz / awesome-panoptic-segmentation

Panoptic Segmentation Resources List

Awesome-Panoptic-Segmentation

This repo is a collection of the challenging panoptic segmentation, including papers, codes, and benchmark results, etc.

Outline

• Panoptic Segmentation
• Datasets
• Evaluation
• Benchmark Results
• Papers
• Tutorials
• Blogs

Panoptic Segmentation

Summarize in one sentence : Panoptic Segmentation proposes to solve the semantic segmentation(*Stuff*) and instance segmentation(*Thing*) in a unified and general manner.

Structure Overview

from UPSNet.

Datasets

Generally, the datasets which contains both semantic and instance annotations can be used to solve the challenging panoptic task.

• COCO-Panoptic
• Cityscapes
• Mapillary Vistas
• ADE20K
• IDD20K

Evaluation

Metrics

• PQ are the standard metrics described in Panoptic Segmentation.

• PC are the standard metrics described in DeeperLab.

Evaluation Code

• cocodataset/panopticapi
• mcordts/cityscapesScripts

Competition

• AutoNUE 2019 Panoptic Segmentation Challenge (ICCV 2019 Workshop, Closed)
• COCO 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)
• Mapillary 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)
• Cityscapes Panoptic Semantic Labeling Task (Open)
• COCO 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)
• Mapillary Vistas 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)

Benchmark Results

COCO val Benchmark

Method	Backbone	PQ	PQ-Thing	PQ-Stuff	SQ	RQ	mIoU	AP-Mask	PC	e2e
SOGNet	ResNet-50	43.7	50.6	33.2	78.7	53.5	54.56	34.2	-	✅
UPSNet	ResNet-50	42.5	48.6	33.4	-	-	54.3	34.3	-	✅
OANet	ResNet-101	41.3	50.4	27.7	-	-	-	-	-	✅
OCFusion	ResNet-50	41.0	49.0	29.0	77.1	50.6	-	-	-	✅
Panoptic FPN	ResNet-101	40.9	48.3	29.7	-	-	-	-	-	✅
AUNet	ResNet-50	39.6	49.1	25.2	-	-	45.1	34.7	-	✅
AdaptIS	ResNet-101	37.0	41.8	29.9	-	-	-	-	-	✅
DeeperLab	Xception-71	34.3	37.5	29.6	77.1	43.1	-	-	56.8	✅

Cityscapes valBenchmark

Method	Backbone	PQ	PQ-Thing	PQ-Stuff	SQ	RQ	mIoU	AP-Mask	PC	e2e
Panoptic(Merge)	-	61.2	66.4	54.0	80.9	74.4	-	-	-	❎
AdaptIS	ResNet-101	60.6	58.7	64.4	-	-	79.2	36.3	-	✅
SOGNet	ResNet-50	60.0	56.7	62.5	-	-	-	-	-	✅
Seamless	ResNet-50	59.8	53.4	64.5	-	-	75.4	31.9	-	✅
UPSNet	ResNet-50	59.3	54.6	62.7	79.7	73.0	75.2	33.3	-	✅
TASCNet	ResNet-101	59.2	56	61.5	-	-	77.8	37.6	-	✅
AUNet	ResNet-101	59.0	54.8	62.1	-	-	75.6	34.4	-	✅
Panoptic FPN	ResNet-101	58.1	52.0	62.5	-	-	75.7	33.0	-	✅
DeeperLab	Xception-71	56.5	-	-	-	-	-	-	75.6	✅

Mapillary val Benchmark

Method	Backbone	PQ	PQ-Thing	PQ-Stuff	SQ	RQ	mIoU	AP-Mask	PC	e2e
Panoptic(Merge)	-	38.3	41.8	35.7	73.6	47.7	-	-	-	❎
Seamless	ResNet-50	37.2	33.2	42.5	-	-	50.2	16.3	-	✅
AdaptIS	ResNet-101	33.4	28.3	40.3	-	-	-	-	-	✅
TASCNet	ResNet-101	32.6	31.3	34.4	-	-	35.0	18.5	-	✅
DeeperLab	Xception-71	32.0	-	-	-	-	-	-	55.3	✅

Papers

AAAI2020

• SOGNet: Yibo Yang, Hongyang Li, Xia Li, Qijie Zhao, Jianlong Wu, Zhouchen Lin.
  "SOGNet: Scene Overlap Graph Network for Panoptic Segmentation." AAAI (2020). [paper]

ICCV2019

• AdaptIS: Konstantin Sofiiuk, Olga Barinova, Anton Konushin.
  "AdaptIS: Adaptive Instance Selection Network." ICCV (2019). [paper]

• Cheng-Yang Fu, Tamara L. Berg, Alexander C. Berg.
  "IMP: Instance Mask Projection for High Accuracy Semantic Segmentation of Things." ICCV (2019). [paper]

• Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen.
  "Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation Bowen." ICCVW (2019). [paper]

CVPR2019

• Panoptic Segmentation: Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár.
  "Panoptic Segmentation." CVPR (2019). [paper]

• Panoptic FPN: Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár.
  "Panoptic Feature Pyramid Networks." CVPR (2019 oral). [paper] [unofficial code][detectron2]

• AUNet: Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang.
  "Attention-guided Unified Network for Panoptic Segmentation." CVPR (2019). [paper]

• UPSNet: Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun.
  "UPSNet: A Unified Panoptic Segmentation Network." CVPR (2019 oral). [paper] [code]

• DeeperLab: Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen.
  "DeeperLab: Single-Shot Image Parser." CVPR (2019 oral). [paper] [project] [code]

• OANet: Huanyu Liu, Chao Peng, Changqian Yu, Jingbo Wang, Xu Liu, Gang Yu, Wei Jiang.
  "An End-to-End Network for Panoptic Segmentation." CVPR (2019). [paper]

• Eirikur Agustsson, Jasper R. R. Uijlings, Vittorio Ferrari .
  "Interactive Full Image Segmentation by Considering All Regions Jointly." CVPR (2019). [paper]

• Seamless: Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, Peter Kontschieder.
  "Seamless Scene Segmentation." CVPR (2019) (Extended Version). [paper][code]

ECCV2018

• Qizhu Li, Anurag Arnab, Philip H.S. Torr.
  "Weakly- and Semi-Supervised Panoptic Segmentation." ECCV (2018). [paper] [code]

ArXiv

• Rohit Mohan, Abhinav Valada.
  "EfficientPS: Efficient Panoptic Segmentation." arXiv (2020). [paper]

• Rui Hou, Jie Li, Arjun Bhargava, Allan Raventos, Vitor Guizilini, Chao Fang, Jerome Lynch, Adrien Gaidon.
  "Real-Time Panoptic Segmentation from Dense Detections." arXiv (2019). [paper]

• Mark Weber, Jonathon Luiten, Bastian Leibe.
  "Single-Shot Panoptic Segmentation." arXiv (2019). [paper]

• Qiang Chen, Anda Cheng, Xiangyu He, Peisong Wang, Jian Cheng.
  "SpatialFlow: Bridging All Tasks for Panoptic Segmentation." arXiv (2019). [paper]

• Sagi Eppel, Alan Aspuru-Guzik.
  "Generator evaluator-selector net: a modular approach for panoptic segmentation." arXiv (2019). [paper]

• Jasper R. R. Uijlings, Mykhaylo Andriluka, Vittorio Ferrari.
  "Panoptic Image Annotation with a Collaborative Assistant." arXiv (2019). [paper]

• OCFusion: Justin Lazarow, Kwonjoon Lee, Zhuowen Tu.
  "Learning Instance Occlusion for Panoptic Segmentation." arXiv (2019). [paper]

• PEN: Yuan Hu, Yingtian Zou, Jiashi Feng.
  "Panoptic Edge Detection." arXiv (2019). [paper]

• TASCNet: Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, Adrien Gaidon.
  "Learning to Fuse Things and Stuff." arXiv (2018). [paper]

• Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
  "Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network." arXiv (2018). [paper]

• Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
  "Single Network Panoptic Segmentation for Street Scene Understanding." arXiv (2019). [paper]

• David Owen, Ping-Lin Chang.
  "Detecting Reflections by Combining Semantic and Instance Segmentation." arXiv (2019). [paper]

• Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji.
  "PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things." arXiv (2019, IROS). [paper]

Tutorials

• CVPR 2019 Tutorial on Visual Recognition and Beyond. [slides] [homepage]
• COCO 2017 Workshop. [slides]

Blogs

• Megvii(Face++) Detection Team. [zhihu]

awesome-panoptic-segmentation's Issues

About your Benchmark Results

In the coco dataset, UPSNET can gain 42.5 PQ with R50 instead of R101 which you can refer to the chapter 4 in their paper.

The content needs to be updated

Please update the content, I'm a freshman in this field, your guys repro give me many advices. Thanks, please update the content. :-)

SQ RQ of SOGNet

The SQ and RQ of SOGNet (R-50) on COCO-val are 78.7 and 53.5, which are shown in Table 2 of the paper.
The mIoU and AP of SOGNet (R-50) on COCO-val are 54. 56 and 34.2, which are described in the last paragraph of the paper.

About your list

are you sure that

Learning to Fuse Things and Stuff and An End-to-End Network for Panoptic Segmentation are accepted by CVPR2019?