zkawfanx / stablellve

Learning Temporal Consistency for Low Light Video Enhancement from Single Images (CVPR2021)

StableLLVE

This is a Pytorch implementation of "Learning Temporal Consistency for Low Light Video Enhancement from Single Images" in CVPR 2021, by Fan Zhang, Yu Li, Shaodi You and Ying Fu.

Paper and Supplemental

Requirements

• python 3.6.9
• Pytorch 1.5.0
• opencv-python 3.4.2

Usage

Training

First, prepare your own traning data and put it in the folder ./data. By default, the code takes input images and ground truth from ./data/train and ./data/gt and you can also change the path in train.py and dataloader.py.

Second, you need to predict plausible optical flow for your ground truth images and put it in the folder ./data/flow. In our paper, we first perform instance segmentation to get object masks using the opensource toolkit detectron2. Then we utilize the pretrained CMP model to generate the optical flow we need.

Update:

• The prediction code and example images now are released in etc folder.
• The noise generation code to be updated.

Finally, you can train models on your own data by running

cd StableLLVE
python train.py 

You can replace the U-Net with your own model for low light image enhancement. The model will be saved in the folder ./logs.

Testing

You can put your test images into the folder ./data/test and just run

cd StableLLVE
python test.py

Model

• checkpoint.pth (This model is trained with the synthetic clean data reported in the paper. It may be unsuitable for noisy data.)

Bibtex

If you find this repo useful for your research, please consider citing our paper.

@InProceedings{Zhang_2021_CVPR,
    author    = {Zhang, Fan and Li, Yu and You, Shaodi and Fu, Ying},
    title     = {Learning Temporal Consistency for Low Light Video Enhancement From Single Images},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {4967-4976}
}

stablellve's Issues

Nice job! and sir can you please release the naive code of predicting the instance segmentation results and flow results?

Nice job!
and sir can you please release the naive code of predicting the instance segmentation results and flow results?
it will help a lot!

How to calculate AB(var) and MABD

Hello, I use your model, but I cannot get the result reported in paper. Can you provide your code for calculating AB(var) and MABD?

全局仿射变换与CMP预测的光流相结合

您能提供全局仿射变换与CMP预测的光流相结合的代码吗？ 非常感谢！！

training dataset

Thank you for your wonderful article! Can you provide training dataset？

Request of Code to Add Noise

Hi,
I am trying to follow your awesome CVPR work. Would you like to share the code for noise generation? Many thanks.
Best regrads,
Jeff

图像质量测评

请问论文中的Table1.是在哪个数据集上进行的指标测评？ 您能提供这个测试数据集吗？我想在这个数据集上对我新训练的模型进行测评

Sir Can you provide your synthetic dataset .It will be much helpful. Thank you

Can't load the checkpoint.pth

Hello, I can't load the checkpoint.pth while testing some images.
Does the pretrained model work?

Thanks a million.

About the Le calculation in training

Thank you for your wonderful article. In the article, you pointed out that the enhancement loss Le includes two parts, the Le of (pred, target) and the Le of (pred_warped, target_warped), but in fact I only see the unwarp Le in train.py.

global affine transformation 全局仿射变换

请问下在您的补充材料中 有提到全局仿射变换，请问这个过程是怎么进行的呢？补充材料中的Figure 2. 的Global那一行的图片是如何得到的？ 非常需要您的指点与解答

关于对比实验

你好！我想请教一下 在你论文的对比实验中，是有MBLLEN 和 MBLLVEN 2种结果，但是我在MBLLEN的源码中并没有找到MBLLVEN的模型，请问你的实验里MBLLVEN的结果是怎么测出来的，MBLLVEN的模型在哪里获得呢？
这是MBLLEN的github地址:https://github.com/Lvfeifan/MBLLEN

When will the paper be released

实验评估

您论文中使用了PSNR、SSIM、AB(Var)、 MABD、WE(×1e-3)进行图像质量的评估，请问是用的什么数据集作为基准来评估的呢？
您能否发布论文中的评估代码呢？感激不尽！！

Can you elaborate on how you are adding poisson noise to the images

测试是否可以输入视频

你好 请问是否可以对自己的低光视频进行测试增强呢？

关于DAVIS数据集生成

我能问一下你们davis中同一段视频中每张图片转成lowlight的参数是相同的，还是说同一段视频中每张图片转成lowlight的参数都不一样

测试结果

下面这2张图是我训练出来的模型得到的最终测试结果，增强后的图片很不自然，很多区域变的很朦胧。我是按照您的论文来处理的训练数据：首先把DAVIS数据集合成低光数据集，然后用detectron2得到掩码，再用得到的掩码进行CMP预测光流。
我唯一缺少的步骤就是没有将CMP预测的光流和全局仿射变换相结合。所以是因为缺少了这个步骤导致最后生成的模型不行吗？您有什么建议或者看法呢

关于SMOID的测试实验结果

SMOID这篇论文利用了一个共轴光学系统建立了一个SMOID的数据集，所以不应该是只能在SMOID这个数据集上测试吗？您是如何在自己创建的DAVIS数据集上测试SMOID模型的？

aout add noise

Is the description of the module you added noise wrong?
poisson_nosie is that

is too much noise

Can you provide pretrained model for noisy data as well

Regarding Optical Flow

Hello @zkawfanx!

In train.py, you use the function WarpingLayerBWFlow. Just wanted to know if the flow predicted using CMP is the backward flow, because warp.py also contains a function named WarpingLayerFWFlow, which seems longer and is able to handle masks as well.

Thanks!