nju-lhrs / ome

One model is enough: Toward multiclass weakly supervised remote sensing image semantic segmentation

One model is enough: Toward multiclass weakly supervised remote sensing image semantic segmentation

Zhenshi Li, Xueliang Zhang, and Pengfeng Xiao

Introduction | Usage| Visual results| Acknowledgement | Statement

Introduction

This is the official repository for the paper “One model is enough: Toward multiclass weakly supervised remote sensing image semantic segmentation”, based on SEAM and mmsegmentation.

Abstract: Semantic segmentation of remote sensing images is effective for large-scale land cover mapping, which heavily relies on a large amount of training data with laborious pixellevel labeling. Weakly supervised semantic segmentation (WSSS) based on image-level labels has attracted intensive attention due to its easy availability. However, existing image-level WSSS methods for remote sensing images mainly focus on binary segmentation, which are difficult to apply to multiclass scenarios. This study proposes a comprehensive framework for image-level multiclass WSSS of remote sensing images, consisting of appropriate image-level label generation, high-quality pixel-level pseudo mask generation, and segmentation network iterative training. Specifically, a training sample filtering method, as well as a dataset cooccurrence evaluation metric, is proposed to demonstrate proper image-level training samples. Leveraging multiclass class activation maps, an uncertainty-driven pixel-level weighted mask is proposed to relieve the overfitting of labeling noise in pseudo masks when training the segmentation network. Extensive experiments demonstrate that the proposed framework can achieve high-quality multiclass WSSS performance with image-level labels, which can attain 94.23% and 90.77% of the IoUs from pixel-level labels for the ISPRS Potsdam and Vaihingen datasets, respectively. Beyond that, for the DeepGlobe dataset with more complex landscapes, the WSSS framework can achieve an accuracy close to 99% of the fully supervised case. Additionally, we further demonstrate that compared to adopting multiple binary WSSS models, directly training a multiclass WSSS model can achieve better results, which can provide new thoughts to achieve WSSS of remote sensing images for multiclass application scenarios.

Usage

Classification network

Several classification networks are adopted in our OME inplement. We show our SEAM-based network here.

1. co-occurrence matrix generation: Myutils/calculate_cooccur_matrix.py
2. SEAM training: train_SEAM.py
3. SEAM inference: infer_SEAM.py
4. uncertainty generation from CAMs: Myutils/generate_uncertainty_from_cam.py

The ImageNet pretrained model weights can be downloaded from SEAM.

Segmentation network

Please refer the segmentation phase to URN based on mmsegmentation, with a improved reweighting operation as in our paper.

Visual results

Results on the ISPRS Potsdam dataset

Results on the ISPRS Vaihingen dataset

Results on the Deepglobe dataset

Acknowledgement

• Many thanks to the following repos: SEAM, URN, and mmsegmentation.

Statement

• Please cite our paper if our work is useful to your research.

    @article{li2023one,
      title={One model is enough: Toward multiclass weakly supervised remote sensing image semantic segmentation},
      author={Li, Zhenshi and Zhang, Xueliang and Xiao, Pengfeng},
      journal={IEEE Transactions on Geoscience and Remote Sensing},
      year={2023},
      publisher={IEEE}
    }
  

• Any questions please contact LZhenShi ([email protected]).

ome's Issues

Questions about the work

Thank you very much for your work, but I have some questions about it. In the code, it seems that you have ignored the background class in the Potsdam dataset. So, when generating pseudo-labels and later training the segmentation model, is the background class also ignored when calculating mIoU against the ground truth labels? Additionally, it seems that the paper and code you provided don't quite match. What's the reason for this?Looking forward to your reply."

segmentation configs

hi could you please provide the training configs used in URN segmentation works?

Iterative training

Author, hello! How is the iterative training of the segmentation network implemented? It appears that there is no code related to it in the provided text.

multi class

Author, I would like to ask about the emphasis on multi-class in your work based on the SEAM code. I didn't see any significant changes related to multi-class in the code. Could you please clarify what multi-class refers to in this context?

accuracy

I have a question, can the accuracy index achieved in the article reach this high?

guess the paper link is wrong

now Im reading this CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding

segmentation network code

Hello author, may I ask if you can provide the code for dividing the network? Even if there is a reference URN, its structure is quite complex. Is it possible to provide segmentation network code specifically for this article?

How did you exclude the dataset? Can you provide the dataset?

Hi! Excellent Work! Saw it from Wechat.
I'm very insterested in you work. However, I'm confused about the filtered dataset, it seems quite important for the training process, but I haven't get it that how did you exclude the dataset. Quoted from the paper that 1) "The patches for training the classification network are filtered by removing the samples with pixels of any category smaller than 10% (2.5% for the small object of car) of the total pixels." 2) "Hence, we exclude samples with a small number of pixels for certain category to train the classification network."
How did you exclude the dataset? Can you provide the dataset? Thanks!