swz30 / cycleisp

[CVPR 2020--Oral] CycleISP: Real Image Restoration via Improved Data Synthesis

CycleISP: Real Image Restoration via Improved Data Synthesis (CVPR 2020 -- Oral)

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao

---

Abstract: The availability of large-scale datasets has helped unleash the true potential of deep convolutional neural networks (CNNs). However, for the single-image denoising problem, capturing a real dataset is an unacceptably expensive and cumbersome procedure. Consequently, image denoising algorithms are mostly developed and evaluated on synthetic data that is usually generated with a widespread assumption of additive white Gaussian noise (AWGN). While the CNNs achieve impressive results on these synthetic datasets, they do not perform well when applied on real camera images, as reported in recent benchmark datasets. This is mainly because the AWGN is not adequate for modeling the real camera noise which is signal-dependent and heavily transformed by the camera imaging pipeline. In this paper, we present a framework that models camera imaging pipeline in forward and reverse directions. It allows us to produce any number of realistic image pairs for denoising both in RAW and sRGB spaces. By training a new image denoising network on realistic synthetic data, we achieve the state-of-the-art performance on real camera benchmark datasets. The parameters in our model are ~5 times lesser than the previous best method for RAW denoising. Furthermore, we demonstrate that the proposed framework generalizes beyond image denoising problem e.g., for color matching in stereoscopic cinema.

CycleISP for Synthesizing Data for Image Denoising

The proposed CycleISP framework allows converting sRGB images to RAW data, and then back to sRGB images. It has (a) RGB2RAW network branch, and (b) RAW2RGB network branch.

Overall Framework of CycleISP

Recursive Residual Group (RRG)

Proposed Denoising Network

Installation

The model is built in PyTorch 1.1.0 and tested on Ubuntu 16.04 environment (Python3.7, CUDA9.0, cuDNN7.5).

For installing, follow these intructions

sudo apt-get install cmake build-essential libjpeg-dev libpng-dev
conda create -n pytorch1 python=3.7
conda activate pytorch1
conda install pytorch=1.1 torchvision=0.3 cudatoolkit=9.0 -c pytorch
pip install matplotlib scikit-image yacs lycon natsort h5py tqdm

Data Synthesis using CycleISP

• Download all the models and place them in ./pretrained_models/isp/
• Download some sample images from the MIR-Flickr dataset and place them in ./datasets/some_sample_images/

Generating Data for RAW Denoising

The RGB2RAW network branch takes as input a clean sRGB image and converts it to a clean RAW image. The noise injection module adds shot and read noise of different levels to the (RAW) output of RGB2RAW network branch. Thereby, we can generate clean and its corresponding noisy image pairs {RAW_clean, RAW_noisy} from any sRGB image.

• Run Demo

python generate_raw_data.py 

Generating Data for sRGB Denoising

Given a synthetic RAW noisy image as input, the RAW2RGB network branch maps it to a noisy sRGB image; hence we are able to generate an image pair {sRGB_clean, sRGB_noisy} for the sRGB denoising problem.

• Run Demo

python generate_rgb_data.py 

Evaluation

You can download, at once, the complete repository of CycleISP (including pre-trained models, datasets, results, etc) from this Google Drive link, or evaluate individual tasks with the following instructions:

Denoising RAW images of DND

• Download the model and place it in ./pretrained_models/denoising/
• Download RAW images of DND and place them in ./datasets/dnd/dnd_raw/
• Run

python test_dnd_raw.py --save_images

Denoising RAW images of SIDD

• Download the model and place it in ./pretrained_models/denoising/
• Download RAW images of SIDD and place them in ./datasets/sidd/sidd_raw/
• Run

python test_sidd_raw.py --save_images

Denoising sRGB images of DND

• Download the model and place it in ./pretrained_models/denoising/
• Download sRGB images of DND and place them in ./datasets/dnd/dnd_rgb/noisy/
• Run

python test_dnd_rgb.py --save_images

Denoising sRGB images of SIDD

• Download the model and place it in ./pretrained_models/denoising/
• Download sRGB images of SIDD and place them in ./datasets/sidd/sidd_rgb/
• Run

python test_sidd_rgb.py --save_images

Results on Real Image Datasets

Experiments are performed for denoising images in RAW and sRGB spaces. Results produced by CycleISP can be downloaded from Google Drive link.

Results for RAW Denoising

Results for sRGB Denoising

Citation

If you use CycleISP, please consider citing:

@inproceedings{Zamir2020CycleISP,
    title={CycleISP: Real Image Restoration via Improved Data Synthesis},
    author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat
            and Fahad Shahbaz Khan and Ming-Hsuan Yang and Ling Shao},
    booktitle={CVPR},
    year={2020}
}

Contact

Should you have any question, please contact [email protected]

Our Related Works

-- Learning Enriched Features for Fast Image Restoration and Enhancement, TPAMI 2022. Paper | Code

• Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 2022. Paper | Code
• Multi-Stage Progressive Image Restoration, CVPR 2021. Paper | Code
• Learning Enriched Features for Real Image Restoration and Enhancement, ECCV 2020. Paper | Code

cycleisp's Issues

unpadding

CycleISP/generate_raw_data.py

Line 80 in 76a52aa

clean_packed = clean_packed[:,padh[j]//2:-padh[j]//2,padw[j]//2:-padw[j]//2] ## RGGB channels (4 x H/2 x W/2)

i don't understand the meaning of the unpadding ops.

Some questions about GPU settings

Hello, I tried to train the DenoiseNet with my own code on an NVIDIA 1080 Ti GPU, but the GPU memory runs out and I can only train it with size of 64 and batch of 1.
I would like to know more training details about the GPU settings, such as GPU numbers, etc.

question about the training dataset

I want to know if you only used raw images to train the RGB2RAW Network Branch and RAW2RGB Network Branch. For example, you used clean sRGB images as ground truth for optimizing RAW2RGB network, and I don't know how to get the ground truth.

Log Files from Training

Thank you for your awesome code!

I am hoping you might open-source the log files you have from training. Maybe the training and validation loss as a function of epoch
(and/or batch) with an estimate of the runtime?

Installation with lycon

We need to install cmake to run lycon. Although it is better to use cv2 than lycon
conda install cmake
pip install lycon

do this if you still want to use lycon

About 'lycon' module

Hi,when I run the 'test_dnd_rgb.py',If I didn't set the Parameter:'--save images',I can ran the code.
However,if I set the Parameter,I met the error:

Traceback (most recent call last):
  File "/workspace/Cycle/test_dnd_rgb.py", line 66, in <module>
    lycon.save(args.result_dir + 'png/'+ filenames[batch][:-4] + '.png', denoised_img)
  File "/opt/conda/lib/python3.7/site-packages/lycon/core.py", line 24, in save
    _lycon.save(path, image, options)
SystemError: <built-in function save> returned a result with an error set

Can u tell me why? Is the error of the 'lycon' version?But I only can pip install the version==0.2.0,the other version can't be pip install.

how to train your model?

can you share your training code?@swz30

training code and dataset

Can you provide the training code and dataset?
Thanks

Training code

Could you please share your code for training model or your loss for training model?

the raw2rgb results have artifacts

I choose adobe fivek_dataset one picture “a3501-dgw_154.dng”， use rawpy to get raw picture（RGBG） and sRGB image（all crop to 512*512）， but the result has artifact。here is the code:

def read_dng():
    with torch.no_grad():
        path = r'E:\MachineLearning_Data\fivek_dataset\raw_photos\HQa3501to4200\photos\a3501-dgw_154.dng'
        raw = rawpy.imread(path)
        raw_img = raw.raw_image_visible
        print(np.max(raw_img))
        width, height = 512, 512
        start_w, start_h = 1000, 1000
        raw_img_save = raw_img[start_h:start_h+height, start_w:start_w+width]
        # raw_img_save.tofile('a3501-dgw_154_crop.raw')
        raw_img_save = pack_raw(np.expand_dims(raw_img_save/(2**14-1), -1))
        raw_img_save = torch.tensor(raw_img_save, dtype=torch.float32).unsqueeze(0).permute(0, 3, 1, 2).cuda()
        img_rgb = raw.postprocess(no_auto_bright=True, user_wb=raw.daylight_whitebalance)
        img_rgb_save = img_rgb[start_h:start_h+height, start_w:start_w+width, :]
        cv2.imwrite('a3501-dgw_154_crop.jpg', img_rgb_save[..., ::-1])
        img_rgb_save = torch.tensor(img_rgb_save/255.0, dtype=torch.float32).permute(2, 0, 1).cuda()
        ccm_tensor = model_ccm(img_rgb_save.unsqueeze(0))
        rgb_noisy = model_raw2rgb(raw_img_save, ccm_tensor)
        rgb_noisy = rgb_noisy.squeeze(0).permute(1, 2, 0)
        cv2.imwrite('raw2rgb_rgbg.jpg', ((rgb_noisy*255.0).cpu().detach().numpy()[..., ::-1]).astype(np.uint8))
[read_dng()]

good work!

Your work is very good, can you provide the training code

Training code

Hello. could u please show us the training part code, thank u~

Question about the SIDD raw results.

Can CycleISP reach 52.41dB PSNR on SIDD benchmark or just on validation set?

Even with the pretrained model, I cannot get that result on the offical server.

About the training code

Do you plan to release the code for the training part?

When is the time to open source code?

Hello
I want to reproduce the results of this thesis through the code you provided. I don't know when you will open source. I hope you open source as soon as possible to help my research. Thank you very much.

training code

Can you provide the training code?Thank you!

rgb2raw branch mosaic problem

In the Rgb2Raw branch，the feature map before Mosaic is HxWx3，and you use this code to generate final raw output:

def mosaic(images):
  shape = images.shape
  red = images[:, 0, 0::2, 0::2]
  green_red = images[:, 1, 0::2, 1::2]
  green_blue = images[:, 1, 1::2, 0::2]
  blue = images[:, 2, 1::2, 1::2]
  images = torch.stack((red, green_red, green_blue, blue), dim=1)
  # images = tf.reshape(images, (shape[0] // 2, shape[1] // 2, 4))
  return images

this just use litter data of images to generate raw, I find some details in your paper in Section3.1,

the Bayer sampling function f_Bayer is applied

So why just output a feature map with shape HxWx1 to full use the data?

SIDD validation's bayer pattern is unknown, how to stack per bayer's data as R-G-G-B sequence.

Thanks for your good work !
And thanks for putting the processed SIDD validation data on Google Drive. I noticed the original SIDD validation's bayer pattern is unknown in the benchmark website, so how did you process them to stack per bayer's data as R_G_G_B sequence?

Raw2Rgb network doesnt match the one in paper

In the forward method of Raw2Rgb module in cycleisp.py, x = self.body【-1】(x) indicates that x is activated by 'nn.PReLU(n_feats)'. As the Figure-2 of paper show, it should be followed by the k-th RRG block here instead of activtion.

def forward(self, x, ccm_feat):
    x = self.head(x)
    for i in range(len(self.body)-1):
        x = self.body[i](x)
    body_out = x.clone()
    x = x * ccm_feat          ## Attention 
    x = x + body_out       
    x = self.body[-1](x)
    x = self.tail(x)
    x = self.tail_rgb(x)
    x = nn.functional.pixel_shuffle(x, 2)
    return x

Questions about the loss function used by the RAW2RGB network

Hello,
I am wondering why not use the log loss term in the RAW2RGB network, just like in the RGB2RAW network. Are there any principles? Or is it just set empirically?

Initial training of CycleISP

Hi,
I'm trying to do the initial training on the MIT-Adobe FiveK dataset. The raw image is huge which makes the loader slow. Did you do the cropping online?
Thanks.

Training code

Any updates on when will you release the training codes?

Training Code

Any updates on when will you release the training codes?

Version update for skimage.measure compare_psnr and compare_ssim

To run the code change "from skimage.measure.simple_metrics import compare_psnr " to " from skimage.metrics import peak_signal_noise_ratio as compare_psnr " and
"from skimage.measure import compare_ssim" to
from skimage.metrics import structural_similarity as compare_ssim
in utils/image_utils.py

These are the new version update for skimage

License information

Hello,

Great work! I was wondering is it possible to add an MIT license to the code?

How to test raw images without gt?

Thanks for your good work!
How to test raw images without gt? Without gt, I can not get the variance, then I can not get the restored image.