sjmoran / deeplpf

Code for CVPR 2020 paper "Deep Local Parametric Filters for Image Enhancement"

DeepLPF: Deep Local Parametric Filters for Image Enhancement (CVPR 2020)

Sean Moran, Pierre Marza, Steven McDonagh, Sarah Parisot, Greg Slabaugh

Huawei Noah's Ark Lab

Main repository for the CVPR 2020 paper DeepLPF: Deep Local Parametric Filters for Image Enhancement. Here you will find a link to the code, pre-trained models and information on the datasets. Please raise a Github issue if you need assistance of have any questions on the research.

Training your own DeepLPF: if you have difficulities, please contact the lead author for assistance. We are responsive and will be glad to assist on any queries regarding the model.

BATCH SIZE: Note this code is designed for a batch size of 1. It needs re-engineered to support higher batch sizes. Using higher batch sizes is not supported currently. To replicate our reported results please use a batch size of 1 only. If you do have a patch for DeepLPF to support higher batch sizes please raise a pull request on this repo and we will integrate.

[Paper] [Poster] [Video] [Supplementary]

Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)
Input	Label	Ours (DeepLPF)

Dependencies

requirements.txt contains the Python packages used by the code.

How to train DeepLPF and use the model for inference

Training DeepLPF

Instructions:

To get this code working on your system / problem you will need to edit the data loading functions, as follows:

1. data.py, lines 248, 256, change the folder names of the data input and output directories to point to your folder names

To train, run the command:

python3 main.py --training_img_dirpath=../adobe5k/ --train_img_list_path=../adobe5k/images_train.txt --valid_img_list_path=../adobe5k/images_valid.txt --test_img_list_path=../adobe5k/images_test.txt

Inference - Using Pre-trained Models for Prediction

The directory pretrained_models contains a set of four DeepLPF pre-trained models on the Adobe5K_DPE dataset, each model output from different epochs. The model with the highest validation dataset PSNR (23.90 dB) is at epoch 424:

• deeplpf_validpsnr_23.378_validloss_0.033_testpsnr_23.904_testloss_0.031_epoch_424_model.pt

This model achieves a PSNR of 23.90dB and an SSIM of 0.911 on the Adobe_DPE image dataset. To inference with this model, follow these instructions:

1. Place the images you wish to infer in a directory e.g. ./adobe5k_dpe/deeplpf_example_test_input/. Make sure the directory path has the word "input" somewhere in the path.
2. Place the images you wish to use as groundtruth in a directory e.g. ./adobe5k_dpe/deeplpf_example_test_output/. Make sure the directory path has the word "output" somewhere in the path.
3. Place the names of the images (without extension) in a text file in the directory above the directory containing the images i.e. ./adobe5k_dpe/ e.g. ./adobe5k_dpe/images_inference.txt
4. Run the command and the results will appear in a timestamped directory in the same directory as main.py:

python3 main.py --inference_img_list_path=./adobe5k_dpe/images_inference.txt --inference_img_dirpath=./adobe5k_dpe/ --checkpoint_filepath=./pretrained_models/adobe_dpe/deeplpf_validpsnr_23.378_validloss_0.033_testpsnr_23.904_testloss_0.031_epoch_424_model.pt

Pre-trained models

• Adobe-DPE: see pretrained_models/adobe_dpe directory. Model trained on the Adobe5K dataset using the splits and image pre-processing outlined in the DeepPhotoEnhancer paper.
• Adobe-UPE: see pretrained_models/adobe_upe directory. Model trained on the Adobe5K dataset using the splits and image pre-processing outlined in the DeepUPE paper. Thanks to Yucheng Lu ([email protected]) for this pre-trained model. The model was applied in this paper.

Bibtex

@InProceedings{Moran_2020_CVPR,
author = {Moran, Sean and Marza, Pierre and McDonagh, Steven and Parisot, Sarah and Slabaugh, Gregory},
title = {DeepLPF: Deep Local Parametric Filters for Image Enhancement},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

Datasets

• Adobe-DPE (5000 images, RGB, RGB pairs): this dataset can be downloaded here. After downloading this dataset you will need to use Lightroom to pre-process the images according to the procedure outlined in the DeepPhotoEnhancer (DPE) paper. Please see the issue here for instructions. Artist C retouching is used as the groundtruth/target. Note, the images must be extracted in sRGB space. Feel free to raise a Gitlab issue if you need assistance with this (or indeed the Adobe-UPE dataset below). You can also find the training, validation and testing dataset splits for Adobe-DPE in the following file. The splits can also be found the the adobe5k_dpe directory in this repository (note these are a best guess at what the orginal splits from the DPE authors might be).

• Adobe-UPE (5000 images, RGB, RGB pairs): this dataset can be downloaded here. As above, you will need to use Lightroom to pre-process the images according to the procedure outlined in the Underexposed Photo Enhancement Using Deep Illumination Estimation (DeepUPE) paper and detailed in the issue here. Artist C retouching is used as the groundtruth/target. You can find the test images for the Adobe-UPE dataset at this link.

License

BSD-3-Clause License

Contributions

We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion.

If you plan to contribute new features, utility functions or extensions to the core, please first open an issue and discuss the feature with us. Sending a PR without discussion might end up resulting in a rejected PR, because we might be taking the core in a different direction than you might be aware of.

Errata for Paper(s) referencing DeepLPF

• Deep Symmetric Network for Underexposed Image Enhancement with Recurrent Attentional Learning
• The results in Fig 1. for DeepLPF in this paper are incorrect. An example of inference for one of the images in Fig 1. is supplied here. For replication, we provide our DeepLPF pre-trained model for their dataset here. The quantitative results in Table 1 for DeepLPF should be: 23.63dB, 0.875 SSIM.
• 29th September 2021: The paper authors have graciously noted this issue and have created an errata to their ICCV paper here. We thank the authors and appreciate their time commitment and close collaboration in re-checking the DeepLPF result in their paper.

deeplpf's Issues

Could you release the training, validation and testing dataset splits for Adobe-DPE

Thank you for your amazing work.
But I'm sorry that the link in the readme is broken:
https://www.cmlab.csie.ntu.edu.tw/project/Deep-Photo-Enhancer/%5BOnline_Demo_Models%5D_Deep-Photo-Enhancer.zip

So could you please release the training, validation and testing dataset splits for Adobe-DPE

Input to filter blocks

Hi,
According to your poster, the elliptical and graduated filters should be taking concat(y_hat_2, backbone_features) as input, yet according to the following code (see starting point), all 3 filter block types (incl. polynomial/cubic) take concat(y_hat_1, backbone_features)

DeepLPF/model.py

Line 997 in 322e9e0

img_cubic = self.cubic_filter.get_cubic_mask(feat, img)

Could you please help clarify?

Thanks!

Test on custom dataset

Hello, Authors! Thanks for your excellent work. I would like to know whether it is possible to test custom images other than the Adobe dataset. In the code, I found that the data loader is just for Adobe.

data.py only have 248 lines

I want to train the DeepLPF again, but I see the data.py only have 248 lines, so where should I modify the path?

In the other hand, I want to know what means of 3 pre-trained models, I think the adobe_dpe is 2250 training images and 500 testing images, adobe_upe is 4500 training images and 500 testing image, but how about distort-and-recover?

Last question, how do you export the MIT-ADOBE-FIVEK datasets, do you have do any operation such as resizeing?

I use fivek datasets last 500 images as testing images which resize to have a long-edge of 512 pixels in your three pre-trained model and get a not very good result (psnr 21.57 in dpe, 22.51 in upe and 20.87 in distort-and-recover)

hello, I am very interested in your work. Could you upload your supplementary material or your codes

Issue in data.py

Line 232 in data.py should be img_id = file.split(".")[0] instead of img_id = file.split("-")[0]

Training giving strange results

Hey **Sean, totally am inspired by your work.

I've turned DeepLPF into device-agnostic code, to run on "cpu" (I'm on M1 mac and "mps" is still unreliable). I've been successful in testing images based on your existing checkpoints, however am getting strange results when training on my own data, and I can't figure out what would be giving this "look". I've followed the training data image prep as per your readme file.
input:

groundtruth:

test:

Any suggestions?

about the traning dataset

The original mit-adobe-fivek dpe if too big for me and as you said"Lightroom to pre-process the images according to the procedure outlined in the DeepPhotoEnhancer (DPE) paper". Can you provide with a processed input dataset?

Varying input image size

Hello,
Could you please explain how DeepLPF handles various image sizes at (1) train time (2) inference/prediction time?
Also if you could point to parts of the code that support that, that would be highly appreciated.
Thanks!

Adobe-UPE dataset

Adobe-UPE dataset can't bt found, could you provide another link please?

Poor performance on MIT Adobe Fivek datasets

Hello, thank you for excellent work! I have met some problem when I trained and tested your model on the dataset mentioned in your paper. I follow the ways mentioned in DPE to process the MIT Adobe FiveK dataset and use their test list for testing. However, I got very poor performance with PSNR 17.49 dB on the FiveK-DPE test dataset.
the following is my processed image paired.

input image

expertC image

Could you release the pre-trained model for testing SID dataset (fuji camera)? Thanks

Wrong loss function in paper/poster?

Hi,
I believe the paper and poster both show a loss function with contradictive parts: the first part (Lab) gets bigger as images are more different, whereas the second part (MS-SSIM) gets bigger as images are more similar.

The code, however, does not make that mistake, and actually the loss is Lab + (1 - MS-SSIM) (not sure why 1 is necessary).

higher batch sizes

Deer author,I want to know when you will update higher batch sizes。

RuntimeError: Given groups=1, weight of size [16, 3, 3, 3], expected input[1, 4, 345, 514] to have 3 channels, but got 4 channels instead

RuntimeError: Given groups=1, weight of size [16, 3, 3, 3], expected input[1, 4, 345, 514] to have 3 channels, but got 4 channels instead

Could you release your pre-trained model?

Hello, I'm very interested in your great work. Could you upload your pre-trained models? Thanks.