tegg89 / srcnn-tensorflow

Image Super-Resolution Using Deep Convolutional Networks in Tensorflow https://arxiv.org/abs/1501.00092v3

SRCNN-Tensorflow

Tensorflow implementation of Convolutional Neural Networks for super-resolution. The original Matlab and Caffe from official website can be found here.

Prerequisites

• Tensorflow
• Scipy version > 0.18 ('mode' option from scipy.misc.imread function)
• h5py
• matplotlib

This code requires Tensorflow. Also scipy is used instead of Matlab or OpenCV. Especially, installing OpenCV at Linux is sort of complicated. So, with reproducing this paper, I used scipy instead. For more imformation about scipy, click here.

Usage

For training, python main.py
For testing, python main.py --is_train False --stride 21

Result

After training 15,000 epochs, I got similar super-resolved image to reference paper. Training time takes 12 hours 16 minutes and 1.41 seconds. My desktop performance is Intel I7-6700 CPU, GTX970, and 16GB RAM. Result images are shown below.

Original butterfly image:
Bicubic interpolated image:
Super-resolved image:

References

• liliumao/Tensorflow-srcnn
  • • I referred to this repository which is same implementation using Matlab code and Caffe model.

• carpedm20/DCGAN-tensorflow
  • • I have followed and learned training process and structure of this repository.

srcnn-tensorflow's Issues

test time is too high

When testing, if I change the size of the input image, the test time on the GPU will be very long, without changing the size of the image, the second start time will be normal; this problem was not found on the CPU; is this a problem for tensorflow?

I get an error when inference, any idea? thanks

guoxiujiang@kdc-imggroup:/projects/SRCNN-Tensorflow-master$ python main.py --is_train False --stride 21
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
{'batch_size': 128,
'c_dim': 1,
'checkpoint_dir': 'checkpoint',
'epoch': 10,
'image_size': 33,
'is_train': False,
'label_size': 21,
'learning_rate': 0.0001,
'sample_dir': 'sample',
'scale': 3,
'stride': 21}
modprobe: FATAL: Module nvidia-uvm not found in directory /lib/modules/4.4.0-53-generic
E tensorflow/stream_executor/cuda/cuda_driver.cc:491] failed call to cuInit: CUDA_ERROR_UNKNOWN
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:140] kernel driver does not appear to be running on this host (kdc-imggroup): /proc/driver/nvidia/version does not exist
[] Reading checkpoints...
[] Load SUCCESS
Testing...
Traceback (most recent call last):
File "main.py", line 46, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "main.py", line 43, in main
srcnn.train(FLAGS)
File "/home/guoxiujiang/projects/SRCNN-Tensorflow-master/model.py", line 112, in train
imsave(merge, os.path.join(os.getcwd(), config.sample_dir))
File "/home/guoxiujiang/projects/SRCNN-Tensorflow-master/utils.py", line 191, in imsave
return scipy.misc.imsave(path, image)
File "/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py", line 195, in imsave
im = toimage(arr, channel_axis=2)
File "/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py", line 287, in toimage
raise ValueError("'arr' does not have a suitable array shape for "
ValueError: 'arr' does not have a suitable array shape for any mode.
guoxiujiang@kdc-imggroup:/projects/SRCNN-Tensorflow-master$

Train is OK, but test with errors

I ran the file in the spyder and I ran the train process at first.
And this is the error

File "D:\scholarship\anaconda\envs\tensorflow\lib\site-packages\absl\flags_flagvalues.py", line 430, in setitem
raise _exceptions.DuplicateFlagError.from_flag(name, self)

DuplicateFlagError: The flag 'epoch' is defined twice. First from F:/F_coding/SRCNN-Tensorflow-master/main.py, Second from tensorflow.python.platform.flags. Description from first occurrence: Number of epoch [15000]

Why the output test image shrink

when i ran the sentence python main.py --is_train False --stride 21, i find that the output shrink from 256256 to 231231, any idea why the heck this happen ?

How to predict ?

I have two questions:
(1)I run this model ,and found after every epoch, the loss is 10125, and can't decrease, even I change the learning rate.

(2)After train, then can load the model, and how to predict or inference a low resolution picture , such as 1080p to a super resolution picture? thanks

arbitrary image size

What is algorithm to increase quality for arbitrary image size?

.

About result

I used your command "python main.py --is_train False --stride 21" for testing, and found a file named "sample/test_image.png" was created. I wonder what is "test_image.png" ? It is a grey image with a low quality. Thus, I wonder is it just a low quality image that your make for testing SRCNN ? And then how can I use SRCNN to make a super resolution result ?

can't seem to create h5 files

I am attempting to test on a different dataset but it seems to fail during the create dataset portion. Do I need to use empty h5 files before use?

How to get three channels image results?

您好，请问这是只重建了Y通道的图像吗?如何根据这个结果重建彩色图呀，谢谢

what the mean of image_size, label_size and stride

if image_size mean the input size of the image,why the value is 33? And the label size is 21? Can anyone help me? Thanks a lot.

Pretrained Model

Dear Tegg,

could you please provide the pretrained model for SRCNN with RGB as input channel? The original paper only provides a model with Y as input.

Best regards
Andreas

train error

tensorflow.python.framework.errors_impl.UnknownError: Failed to rename: checkpoint\srcnn_21\checkpoint.tmpb873d260b1e14e919dbd7e5b9c1334d8 to: chec kpoint\srcnn_21\checkpoint : \udcbeܾ\udcf8\udcb7\udcc3\udcceʡ\udca3

why？？？

3 channel of RGB problem

I trained the image in RGB format of 3 channels, but the color distortion of the tested image is very serious. Has anyone trained the same and got good results?

want to enlarge the image by 4x

How the model should be set up
want to enlarge the image by 4x

training error

When I am training the model, it appear this error to me:

An exception has occurred, use %tb to see the full traceback.

SystemExit

/anaconda3/envs/py35/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3299: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

then I use %tb to see the full traceback, it print like this:

Traceback (most recent call last):

File "", line 47, in
tf.app.run()

File "/anaconda3/envs/py35/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))

SystemExit

so how to fix this problem?

test_image problem

I set the 'is_train' to False, but I find the size of 'test_image.png' is 336*336 and it seems the test image resolution is very low .
Can anyone help me?

No test result

when i ran the sentence, python main.py --is_train False --stride 21, I COULD only find one image in the file sample, i am not sure where the SR result is ...

Changing filter size.

About trainning

After one training session, do I need to delete checkpoint and train.h5 files to be ready for the next one？

About result and trainning

Can you help me? I calculate your original image and SRCNN image PSNR is only 13. And my end result seemed to be the result of training 100 epochs. PSNR is only 11.92.

Loss drop little

It seems that loss drop limit during training, just 0.03% improve.

How can we make loss drop more quickly?

Different PSNR values from the paper

After training and testing it, I'm getting a different PSNR value which is lower than the one reported from the original paper. Is anyone able to replicate the same PSNR?

Is this a grayscale map only? What do you have to do with a color map?

Apply image file with bicubic interpolation

May I ask why the two line is mean bicubic interpolation? Are there any reference?
https://github.com/tegg89/SRCNN-Tensorflow/blob/master/utils.py#L53-L54
Thanks.

I don't understand the meaning of this code

I guess this code is to make the image blurred
input_ = scipy.ndimage.interpolation.zoom(label_, (1./scale), prefilter=False)
input_ = scipy.ndimage.interpolation.zoom(input_, (scale/1.), prefilter=False)

But after I code like this, I find the image(channel = 3) become darker than before. Are there any problems?
image = scipy.ndimage.interpolation.zoom(image, (1./scale, 1./scale, 1), prefilter=False)
image = scipy.ndimage.interpolation.zoom(image, (scale/1., scale/1., 1), prefilter=False)

The train is ok, but when I run "python main.py --is_train False --stride 21", there is an error.

kedaxtb@kedaxtb:~/guoxiujiang/SRCNN-Tensorflow-master$ python main.py --is_train False --stride 21
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
{'batch_size': 128,
'c_dim': 1,
'checkpoint_dir': 'checkpoint',
'epoch': 10,
'image_size': 33,
'is_train': False,
'label_size': 21,
'learning_rate': 0.0001,
'sample_dir': 'sample',
'scale': 3,
'stride': 21}
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:02:00.0
Total memory: 11.90GiB
Free memory: 11.62GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x30b7230
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties:
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:03:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
WARNING:tensorflow:From /home/kedaxtb/guoxiujiang/SRCNN-Tensorflow-master/model.py:76 in train.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer instead.
[] Reading checkpoints...
[] Load SUCCESS
Testing...
Traceback (most recent call last):
File "main.py", line 46, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "main.py", line 43, in main
srcnn.train(FLAGS)
File "/home/kedaxtb/guoxiujiang/SRCNN-Tensorflow-master/model.py", line 112, in train
imsave(merge, os.path.join(os.getcwd(), config.sample_dir))
File "/home/kedaxtb/guoxiujiang/SRCNN-Tensorflow-master/utils.py", line 191, in imsave
return scipy.misc.imsave(path, image)
File "/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py", line 195, in imsave
im = toimage(arr, channel_axis=2)
File "/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py", line 287, in toimage
raise ValueError("'arr' does not have a suitable array shape for "
ValueError: 'arr' does not have a suitable array shape for any mode.

There is sth that I think it is wrong in this code.

'''
input_ = scipy.ndimage.interpolation.zoom(label_, .5, prefilter=False)
input_ = scipy.ndimage.interpolation.zoom(input_, 2, prefilter=False)
'''

.5 and 2???? I think they should be 1/3. and 3. Because the scale is 3.

By the way, this code helps me a lot.

Is the learning rate the same as mentioned in paper?

I found that the learning rate mentioned in paper is 1e-4 for first two layers and 1e-5 for the last layer and they find that a smaller learning rate in the last layer is important for the network to converge. But I didn't see the change of rated between different layers in this code. Waiting for your reply, thanks!

How to use this code to achieve clarity

Why isthe size

psnr

I can't measure the PSNR in the paper with this code, and the PSNR in Bicubic is also different from that in the paper. Can you give me some comments@tegg89

H5 File doesn't shuffle

when generating h5 files both in training and testing, the order of training samples still remains the order of extracting from images.
In conventional implementation of SRCNN's MATLAB code, they shuffled the training samples before training.

Help plz

Is there anyone can tell me how to use my own train set for this algorithm?

Computation of MSE

In model.py, line 56.

# Loss function (MSE)
lf.loss = tf.reduce_mean(tf.reduce_sum(tf.square(self.labels - self.pred), reduction_indices=0))

It seems not the definition of MSE. How about

lf.loss = tf.reduce_mean(tf.square(self.labels - self.pred))

Thanks.

the Training set

I am sorry,I can't find the training set directory which you use . Can you help. Thanks

how about the result?

I have downloaded your code, and run it directly. It went well. But the result prediction is not the same as your proposal.

How to change which file is being tested??

I attempted to test on the image using the command in the readme. Although the results are poor, I was wondering how to change which file to be tested one. Such as, if I wanted to apply the SRCNN to an image of my choosing.

how to testing and output image

I have run the
python main.py and
python main.py --is_train False --stride 21

no error output, I am wondering how to input an image then generate a new clear image?

about result

I have a question, after this code is finished, only the test files will be generated in sample, so how is the three diagram of the result folder got, I do not get the three pictures, I hope to get help, thank you.

test error

Excuce me, could you help me solve a problem? when I have run the
python main.py , it was ok. But when l have run
`python main.py --is_train False --stride 21 ,it could not run test code.

The result is shown below:
(tf_env) songhuihui@node2:/usrs/yaolu/tf/reproSRCNN/tegg$ python main.py --is_train False --stride 21
{'batch_size': 128,
'c_dim': 1,
'checkpoint_dir': 'checkpoint',
'epoch': 15000,
'image_size': 33,
'is_train': False,
'label_size': 21,
'learning_rate': 0.0001,
'sample_dir': 'sample',
'scale': 3,
'stride': 21}
2019-02-19 20:56:43.642299: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-02-19 20:56:43.769863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:82:00.0
totalMemory: 11.93GiB freeMemory: 1.51GiB
2019-02-19 20:56:43.769918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:82:00.0, compute capability: 5.2)
WARNING:tensorflow:From /home/songhuihui/anaconda3/envs/tf_env/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py:107: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer instead.
[] Reading checkpoints...
[] Load SUCCESS
Testing...
(tf_env) songhuihui@node2:/usrs/yaolu/tf/reproSRCNN/tegg$

what can i do? thanks a lot

IOError: Unsupported BMP header type(108)

show PSNR values

Hi, may I know how to show the super resolution PSNR values?

Thanks! :)

Could you please provide the result image of the degradation model?

Padding

I have discovered something about testing. You have to pad your left and top of your label image to be 6 and you can calculate the rest part. The result psnr will be better and closer to the paper. The setting is like:

height_padding = {} ; width_padding = {}
height_padding['top'] = 6
height_padding['buttom'] = int(abs(result.shape[0] - label.shape[0]) - height_padding['top'])
width_padding['left'] = 6
width_padding['right'] = int(abs(result.shape[1] - label.shape[1]) - width_padding['left'])

ValueError: 'arr' does not have a suitable array shape for any mode.

How to fix it?

$ python main.py --is_train False --stride 21
...
 [*] Load SUCCESS
Testing...
Traceback (most recent call last):
  File "main.py", line 46, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "main.py", line 43, in main
    srcnn.train(FLAGS)
  File "/home/user/SRCNN-Tensorflow/model.py", line 115, in train
    imsave(image_path, result)
  File "/home/user/SRCNN-Tensorflow/utils.py", line 191, in imsave
    return scipy.misc.imsave(path, image)
  File "/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py", line 195, in imsave
    im = toimage(arr, channel_axis=2)
  File "/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py", line 287, in toimage
    raise ValueError("'arr' does not have a suitable array shape for "
ValueError: 'arr' does not have a suitable array shape for any mode.

Why is the size of "test_image" different from the original picture

The size of the "test_image" is 231231 while the size of the original "butterfly_GT" is 256256.

i have a doubt with result

Does checkpoint save your training results? When i run python main.py --is_train False directly , should I get the result as you demonstrate?

A question about image size, label size, stride setting

Dear Author:
I want to ask, how do you set your parameters (image size=33, label size=21, stride=14)?
I have a new training dataset want to use your code to execute. There are 64 images in my new training dataset. All of them original pixel are 282×375. All of test image pixels are 282*375. There are six images in my test dataset.
Now, I have problem in image size, label size, stride setting. Can I have a discussion with you.?
My email is [email protected].
Sincerely,
Chun-Yuan Shin

Do you have to use DCGAN with super-resolution to do something

Hi ,Do you have to use DCGAN with super-resolution to do something?