kunzhan / dssn

ACM MM 2023: Improving semi-supervised semantic segmentation with dual-level Siamese structure network

DSSN

Semi-supervised semantic segmentation (SSS) is an important task that utilizes both labeled and unlabeled data to reduce expenses on labeling training examples. However, the effectiveness of SSS algorithms is limited by the difficulty of fully exploiting the potential of unlabeled data. To address this, we propose a dual-level Siamese structure network (DSSN) for pixel-wise contrastive learning. By aligning positive pairs with a pixel-wise contrastive loss using strong augmented views in both low-level image space and high-level feature space, the proposed DSSN is designed to maximize the utilization of available unlabeled data. Additionally, we introduce a novel class-aware pseudo-label selection strategy for weak-to-strong supervision, which addresses the limitations of most existing methods that do not perform selection or apply a predefined threshold for all classes. Specifically, our strategy selects the top high-confidence prediction of the weak view for each class to generate pseudo labels that supervise the strong augmented views. This strategy is capable of taking into account the class imbalance and improving the performance of long-tailed classes. Our proposed method achieves state-of-the-art results on two datasets, PASCAL VOC 2012 and Cityscapes, outperforming other SSS algorithms by a significant margin.

Getting Started

Installation

Pascal VOC

pip install -r env181.txt

Cityscapes

pip install -r env200.txt

Parts of this code are borrowed from the baseline ST++

Pretrained Backbone

ResNet-50 | ResNet-101 | Xception-65

├── ./pretrained
    ├── resnet50.pth
    ├── resnet101.pth
    └── xception.pth

Dataset

• Pascal: JPEGImages | SegmentationClass
• Cityscapes: leftImg8bit | gtFine
• COCO: train2017 | val2017 | masks

Please modify your dataset path in configuration files.

The groundtruth masks have already been pre-processed by us. You can use them directly.

├── [Your Pascal Path]
    ├── JPEGImages
    └── SegmentationClass
    
├── [Your Cityscapes Path]
    ├── leftImg8bit
    └── gtFine
    
├── [Your COCO Path]
    ├── train2017
    ├── val2017
    └── masks

Usage

DSSN

# use torch.distributed.launch
sh scripts/train.sh <num_gpu> <port>

# or use slurm
# sh scripts/slurm_train.sh <num_gpu> <port> <partition>

To train on other datasets or splits, please modify dataset and split in train.sh.

Supervised Baseline

Modify the method from 'DSSN' to 'supervised' in train.sh, and double the batch_size in configuration file if you use the same number of GPUs as semi-supervised setting (no need to change lr).

Citation

If you find this project useful, please consider citing:

@InProceedings{DSSN2023,
  author    = {Tian, Zhibo and Zhang, Xiaolin and Zhang, Peng and Zhan, Kun},
  booktitle = {ACM Multimedia},
  title     = {Improving semi-supervised semantic segmentation with dual-level Siamese structure network},
  year      = {2023},
}

dssn's Issues

Question about the loss

Hi! In this line of code, MSE Loss is calculated between pred_u_s1_norm and pred_u_s2_norm. However, since two cutmix boxes are different, img_u_s1 and img_u_s2 are definitely different. So how does it make sense to calculate the MSE Loss between two kinds of predictions? Thanks!

Requirement of GPU

Hi! May I ask how many GPUs and what kind of GPU are needed in your work? Thanks!

Question about the difference between cityscapes and pascal voc of the code

Hi! May I ask why you set two different folders of two dataset? I think they can share the same code under two dataset settings. Thanks!

Question about the Pascal blender 1_2 split

Hi! Could you please provide the Pascal blender 1_2 split? I can only see 1_16, 1_8, 1_4 splits of Pascal blender dataset.

What's the difference between your work and unimatch?

I found your code is almost the same as unimatch, but you don't cite unimatch.

Question about the split

Hi! Are the experimental results generated by the splits in U2PL?

Question about the mean-teacher setting

Hi! Have you done the ablation experiment of adopting mean-teacher to provide pseudo labels? In my opinion after various image and feature perturbations, it is no need to adopt MT, the model itself, with applying the classwise-aware threshold, can strong enough to dig out information and generate reliable pseudo labels. I think the performance boosting will be mild, am I right? Thanks!