w1oves / dtp

[ICCV 2023] Official implement of <Disentangle then Parse: Night-time Semantic Segmentation with Illumination Disentanglement>

DTP (Disentangle then Parse)

This repository implements a PyTorch re-implementation of the research paper: "Disentangle then Parse: Night-time Semantic Segmentation with Illumination Disentanglement".

Dataset

Nightcity-fine: Access the dataset via Google Drive.

Cityscapes: Access the dataset via cityscapes-dataset.com.

Environment Setup

Set up your environment with these steps:

conda create -n dtp python=3.10
conda activate dtp
conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia
# Alternatively: pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu116
pip install tensorboard
pip install -U openmim
mim install mmcv-full
pip install -v -e .
# Alternatively: python setup.py develop

Preparation

Download and Organize

1. Decompress the nightcity-fine.zip dataset and relocate it to ./data/nightcity-fine.

2. Download Cityscapes

   1. gtFine_trainvaltest.zip (241MB) [md5]

   2. gtCoarse.zip (1.3GB) [md5]

   3. leftImg8bit_trainvaltest.zip (11GB) [md5]

   4. git clone https://github.com/mcordts/cityscapesScripts.git
      
      pip install cityscapesscripts

   5. Extract all the above zip files and git repo into the ./data folder

   6. vim cityscapesScripts/cityscapesscripts/preparation/createTrainIdLabelImgs.py 
      
      Add the next line of code after `import os`
      os.envieron['CITYSCAPES_DATASET'] = "../../../"
      
      python cityscapesScripts/cityscapesscripts/preparation/createTrainIdLabelImgs.py
      
      mkdir cityscapes
      
      mv gtFine cityscapes && mv leftImg8bit cityscapes

3. Download the checkpoint from Google Drive and place it in ./checkpoints.

Your directory structure should resemble:

.
├── checkpoints
│   ├── night
│   ├── night+day
|   └── simmim_pretrain__swin_base__img192_window6__800ep.pth
├── custom
├── custom-tools
│   ├── dist_test.sh
│   ├── dist_train.sh
│   ├── test.py
│   └── train.py
├── data
│   ├── cityscapes
│   │   ├── gtFine
│   │   └── leftImg8bit
│   └── nightcity-fine
│       ├── train
│       └── val
├── mmseg
├── readme.md
├── requirements.txt
├── setup.cfg
└── setup.py

Testing

Execute tests using:

python custom-tools/test.py checkpoints/night/cfg.py checkpoints/night/night.pth --eval mIoU --aug-test

Training

1. Download pre-training weight from Google Drive.
2. Convert it to MMSeg format using:

   python custom-tools/swin2mmseg.py </path/to/pretrain> checkpoints/simmim_pretrain__swin_base__img192_window6__800ep.pth

3. Start training with:

   python custom-tools/train.py </path/to/your/config>
   # </path/to/your/config>：our config: checkpoints/night/cfg.py or checkpoints/night+day/cfg.py

Results

The table below summarizes our findings:

logs	train dataset	validation dataset	mIoU
checkpoints/night/eval_multi_scale_20230801_162237.json	nightcity-fine	nightcity-fine	64.2
checkpoints/night+day/eval_multi_scale_20230809_170141.json	nightcity-fine + cityscapes	nightcity-fine	64.9

Acknowledgements

This dataset is refined based on the dataset of NightCity by Xin Tan et al. and NightLab by Xueqing Deng et al..

This project is based on the mmsegmentation.

Pretraining checkpoint comes from the SimMIM.

The annotation process was completed using LabelMe.

Citation

If you find this code or data useful, please cite our paper

@InProceedings{Wei_2023_ICCV,
    author    = {Wei, Zhixiang and Chen, Lin and Tu, Tao and Ling, Pengyang and Chen, Huaian and Jin, Yi},
    title     = {Disentangle then Parse: Night-time Semantic Segmentation with Illumination Disentanglement},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {21593-21603}
}

dtp's Issues

When will the code be released ?

A KeyError has occurred

This problem may be my personal problem, but I can't find a solution for the moment because now I haven't other machine.

• envirment prepare:

mim list output:
mmcls      0.25.0
mmcv-full  1.7.2
***
pip list output(key package):
torch                  1.13.1+cu116
torchaudio             0.13.1+cu116
torchvision            0.14.1+cu116
mmsegmentation         0.30.0

---

• problem description：Firstly，I ran the code

python custom-tools/swin2mmseg.py ./premodel/simmim_pretrain__swin_base__img192_window6__800ep.pth ./checkpoints/simmim_pretrain__swin_base__img192_window6__800ep.pth

error output:

.../python3.10/site-packages/mmcv/utils/registry.py", line 61, in build_from_cfg
    raise KeyError(
KeyError: 'CustomTensorboardLoggerHook is not in the hook registry'

---

• detailed description:
  I executed the readme.md step in your code without any errors (or solved it myself), and I was able to run the test.py part of your code as normal.
  I checked all the possibilities I could think of for this problem, including lowering the mvcc-full version to 1.7.0/1.7.1, But it's obviously useless. The only possibility I can think of is that there are some environment version conflicts. Because the main cuda version on my machine was 12.3 and could not be easily changed, I used the pip installation whl package scenario to install the torch environment that met the project requirements.
  So I posed this question to ask if you have encountered this problem while running an existing version of this project, or if you can give me some advice, thank you very much

---

Finally，I think the checkpoints/ & cityscapes/ section of your existing readme.md file structure can be modified. As well as explaining the data generation process for cityscapes (which I inferred from your mention of datasets in your preprint on arxiv), later I hope I can propose a PR for readme.md to your project

Environment Setup error

When I run

pip install -v -e .
or
python setup.py develop

setup.py will push an error :

 File "/data/workspace/ray/xiaomai/DTP/setup.py", line 12, in readme
      with open('MMSEG_README.md', encoding='utf-8') as f:
  FileNotFoundError: [Errno 2] No such file or directory: 'MMSEG_README.md'

I change the code MMSEG_README.md to README.md
New error had pushed ,
Checking requirements.txt shows that the requirements folder is missing from the repository