siavashbigdeli / dmsp

Deep Mean-Shift Priors for Image Restoration

Caffe and MatCovNet implementations (see DMSP-tensorflow for TensorFlow V1 and DMSP-TF2 page for Tensorflow V2 implementation with super-resolution support)

Deep Mean-Shift Priors for Image Restoration (project page)

Siavash Arjomand Bigdeli, Meiguang Jin, Paolo Favaro, Matthias Zwicker

Advances in Neural Information Processing Systems (NIPS), 2017

Abstract:

In this paper we introduce a natural image prior that directly represents a Gaussian-smoothed version of the natural image distribution. We include our prior in a formulation of image restoration as a Bayes estimator that also allows us to solve noise-blind image restoration problems. We show that the gradient of our prior corresponds to the mean-shift vector on the natural image distribution. In addition, we learn the mean-shift vector field using denoising autoencoders, and use it in a gradient descent approach to perform Bayes risk minimization. We demonstrate competitive results for noise-blind deblurring, super-resolution, and demosaicing.

See manuscript for details of the method.

This code runs in Matlab and you need to install either MatCaffe or MatConvNet.

Contents:

demo.m: Includes an example for non-blind and noise-blind image deblurring.

DMSPDeblur.m: Implements MAP function for non-blind image deblurring. Use Matlab's help function to learn about the input and output arguments.

DAEs: Includes DAE models and function handles (in Caffe and matconvnet).

data: Includes sample image and blur kernels.

dmsp's Issues

question about eq.(12) in your paper.

I am confused about eq.(12) in your paper.
What is the gradient operator w.r.t? x or eta?
In context, it should be w.r.t x.
However, I think if it is w.r.t x, the first equal-sign looks false.
Can you explain it in detail? Thanks.

Error in implementing code for Matconvnet in Gpu

I am trying to implement the code for MatConvNet Gpu version.
MATLAB version : R2016b
MatConvNet Version : matconvnet-1.0-beta25
Cuda Version - 8.0
I am getting the following error while implementing current file in github.

I have tried changing
net.layers = net
to net1.layers = net
the code runs for one iteration and then again there is some error.

Please help me sort this error. Thanks.

Edit -

I guess the problem is with the pretrained file format- DAE.sigmall.mat
The net loaded from that file have a different format where we don't actually have layers,

When we add this line
<net.layers = net>

A reference to the same net file is created like this

Which is not the layers that we are trying to access later in code block vl_simplenn_move.
I followed the same code with a different trained model given in matconvnet website. The net format looks like this

Here as we can see each layer can be accessed here.

I still don't know how to use the pre-trained file in current format for the code.

Kernel Blind implementation

I am trying to implement kernel blind from your paper. I was following the Table 1 in the paper for Kernel blind. I am not sure what $x^T$ is doing in 4. equation. Is it just multiplication or convolution? In either case the size of matrix is not satisfied. We have to get a kernel as the final vt but dimensions of term $x^T(K^{t-1}x^{t-1} -y)$ is of (width x height x 3) which we need to convert to kernel which will be (width x height). I am not really sure about the update approach.

As the size of v^t must be equal to the size of the kernel. I don't know how to handle the first term in 4. to maintain the size of the kernel.