weijun88 / ldf

Codes for the CVPR2020 paper "Label Decoupling Framework for Salient Object Detection"

Label Decoupling Framework for Salient Object Detection

by Jun Wei, Shuhui Wang, Zhe Wu, Chi Su, Qingming Huang, Qi Tian

Introduction

To get more accurate saliency maps, recent methods mainly focus on aggregating multi-level features from fully convolutional network (FCN) and introducing edge information as auxiliary supervision. Though remarkable progress has been achieved, we observe that the closer the pixel is to the edge, the more difficult it is to be predicted, because edge pixels have a very imbalance distribution. To address this problem, we propose a label decoupling framework (LDF) which consists of a label decoupling (LD) procedure and a feature interaction network (FIN). LD explicitly decomposes the original saliency map into body map and detail map, where body map concentrates on center areas of objects and detail map focuses on regions around edges. Detail map works better because it involves much more pixels than traditional edge supervision. Different from saliency map, body map discards edge pixels and only pays attention to center areas. This successfully avoids the distraction from edge pixels during training. Therefore, we employ two branches in FIN to deal with body map and detail map respectively. Feature interaction (FI) is designed to fuse the two complementary branches to predict the saliency map, which is then used to refine the two branches again. This iterative refinement is helpful for learning better representations and more precise saliency maps. Comprehensive experiments on six benchmark datasets demonstrate that LDF outperforms state-of-the-art approaches on different evaluation metrics.

Prerequisites

• Python 3.5
• Pytorch 1.3
• OpenCV 4.0
• Numpy 1.15
• TensorboardX
• Apex

Clone repository

git clone https://github.com/weijun88/LDF.git
cd LDF/

Download dataset

Download the following datasets and unzip them into data folder

• PASCAL-S
• ECSSD
• HKU-IS
• DUT-OMRON
• DUTS
• THUR15K

Training & Evaluation

• If you want to train the model by yourself, please download the pretrained model into res folder
• Split the ground truth into body map and detail map, which will be saved into data/DUTS/body-origin and data/DUTS/detail-origin

    python3 utils.py

• Train the model and get the predicted body and detail maps, which will be saved into data/DUTS/body and data/DUTS/detail

    cd train-coarse/
    python3 train.py
    python3 test.py

• Use above predicted maps to train the model again and predict final saliency maps, which will be saved into eval/maps/LDF folder.

    cd /train-fine/
    python3 train.py
    python3 test.py

• Evaluate the predicted results.

    cd eval
    matlab
    main

• Training twice is to get smoother body and detail maps, as shown in following figure

Testing & Evaluate

• If you just want to evaluate the performance of LDF without training, please download our trained model into train-fine/out folder
• Predict the saliency maps

    cd train-fine
    python3 test.py

• Evaluate the predicted results

    cd eval
    matlab
    main

Saliency maps & Trained model

• saliency maps: Google | Baidu 提取码:pb7i

• trained model: Google | Baidu 提取码:sncf

• Quantitative comparisons

• Qualitative comparisons

Citation

• If you find this work is helpful, please cite our paper

@InProceedings{CVPR2020_LDF,
    author    = {Wei, Jun and Wang, Shuhui and Wu, Zhe and Su, Chi and Huang, Qingming and Tian, Qi},
    title     = {Label Decoupling Framework for Salient Object Detection},
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2020}
}

ldf's Issues

accuracy of a salient object detection model

How to get an accuracy of a salient object detection model?

Is there any bounding box based salient object detect?

I want detect generic object in a picture regradless of classes of the object such as SS(Selective Search), however I have not found any article talk about it in recent 3 year, is there anything wrong?

Question.

Firstly, thanks for your contributions. I want to know why is the training part divided into "train-coarse" and "train-fine"? And using the result of the "train-coarse" phase to train "train-fine"? Could you please help me understand that? Thanks a lot!

运行train-fine里的test.py出现如下问题

作者您好，我前面的train-course里的train.py和test.py, train-fine里的train.py都可以运行成功。只有这个出现了报错。我不知道如何解决。能帮忙看看是哪里出问题了？谢谢~~~~

When will the code be released?

Hi weijun, congratulations! I'm really interested in your code and can't wait to try it out. So when are you going to release the code? Thank you! The paper was great!

Can't train the model because of model's apex error

Traceback (most recent call last):
File "/content/drive/MyDrive/Colab Notebooks/LDF/LDF/train-coarse/apex/apex/parallel/init.py", line 15, in
import syncbn
ModuleNotFoundError: No module named 'syncbn'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 15, in
from apex import apex
File "/content/drive/MyDrive/Colab Notebooks/LDF/LDF/train-coarse/apex/apex/init.py", line 8, in
from . import parallel
File "/content/drive/MyDrive/Colab Notebooks/LDF/LDF/train-coarse/apex/apex/parallel/init.py", line 18, in
from .sync_batchnorm import SyncBatchNorm
File "/content/drive/MyDrive/Colab Notebooks/LDF/LDF/train-coarse/apex/apex/parallel/sync_batchnorm.py", line 5, in
from .sync_batchnorm_kernel import SyncBatchnormFunction
File "/content/drive/MyDrive/Colab Notebooks/LDF/LDF/train-coarse/apex/apex/parallel/sync_batchnorm_kernel.py", line 4, in
from apex.parallel import ReduceOp
ModuleNotFoundError: No module named 'apex.parallel'

What wrong to this apex ?

saliency maps

Your work is valuable and interesting. However, you only provide the results under the metrics of MAE, mean-F measure and E-measure. So I want to learn from your saliency maps. Thanks a lot.

recurrent

你好！我最近在复现你的论文，按你的readme中训练过程进行了重新训练，与论文中实验结果有些不同，不知道你在训练时有什么特殊技巧或设置吗。

A typo in your paper.

The "iteration decoder" in Figure.2. should be "iteration encoder".

Inconsistent results on PASCAL-S

Hi,
I just found your provided saliency maps do not have the same evaluation results on PASCAL-S dataset as reported in the paper. I binarize the PASCAL-S dataset for evaluation.
Did you process the PASCAL-S dataset differently?

IOU computing way

hello， In your paper, IOU loss cannot be applied when calculating detail and body, because the input of IOU label is in binary form. But I think BASNet is the IOU loss calculated directly using the true saliency probability. How do you understand this? and the target form is 0 or 1 ? or [0,1] saliency probability

code about utils to generate body and detail map

As you written in your paper,

In addition, we multiply the
newly generated labels with the original binary image I to
remove the background interference as

But I see no where you do this in the code you published, is there something I missed?
Looking for your reply ASAP.

train.txt

你好，
感谢你做的贡献，我想请问一下你训练的数据集是怎么样的？train.txt里面内容是什么？因为我下DUTS数据集训练的时候，一直给我报错‘TypeError: 'NoneType' object is not callable’。
谢谢

EOFError: Ran out of input

Hello,

I have been trying to get your model up and running. I want to test it with the ECSSD Dataset.

I am running Windows 10 with Cuda 10.1.

I get the following error when running: python test.py

Parameters...
datapath  : ../data/ECSSD
snapshot  : ./out/model-40.pt
mode      : test
Traceback (most recent call last):
  File "test.py", line 49, in <module>
    t.save()
  File "test.py", line 35, in save
    for image, (H, W), name in self.loader:
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 801, in __init__
    w.start()
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: 'NoneType' object is not callable

C:\Users\Seppi\Desktop\LDF\train-fine>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\Seppi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input