simonsan / sftgan

CVPR18 - Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform

SFTGAN [Paper] [BasicSR]

😃 Training codes are in BasicSR repo.

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform

By Xintao Wang, Ke Yu, Chao Dong, Chen Change Loy.

This repo only provides simple testing codes - original torch version used in the paper and a pytorch version. For full training and testing codes, please refer to BasicSR.

BibTeX

@InProceedings{wang2018sftgan,
    author = {Wang, Xintao and Yu, Ke and Dong, Chao and Loy, Chen Change},
    title = {Recovering realistic texture in image super-resolution by deep spatial feature transform},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2018}
}

Table of Contents

1. Quick Test
2. Spatial Feature Modulation
3. Semantic Categorical Prior
4. OST dataset

Quick Test

It provides Torch and PyTorch versions. Recommend the PyTorch version.

PyTorch Dependencies

• Python 3
• PyTorch >= 0.4.0
• Python packages: pip install numpy opencv-python

[OR] Torch Dependencies

• Torch
• Other torch dependencies, e.g. nngraph, paths, image (install them by luarocks install xxx)

Test models

Note that the SFTGAN model is limited to some outdoor scenes. It is an unsatisfying limitation that we need to relax in future.

1. Clone this github repo.

git clone https://github.com/xinntao/SFTGAN
cd SFTGAN

2. There are two sample images in the ./data/samples folder.
3. Download pretrained models from Google Drive or Baidu Drive. Please see model list for more details.
4. First run segmentation test.

[PyTorch]

cd pytorch_test
python test_segmentation.py

[Torch]

cd torch_test
th test_segmentation.lua

The segmentation results are then in ./data with _segprob, _colorimg, _byteimg suffix.

5. Run sftgan test.

[PyTorch]

python test_sftgan.py.

[Torch]

th test_sftgan.lua

The results are in then in ./data with _result suffix.

Spatial Feature Modulation

SFT - Spatial Feature Transform (Modulation).

A Spatial Feature Transform (SFT) layer has been proposed to efficiently incorporate the categorical conditions into a CNN network.

There is a fantastic blog explaining the widely-used feature modulation operation distill - Feature-wise transformations.

Semantic Categorical Prior

We have explored the use of semantic segmentation maps as categorical prior for SR.

OST dataset

• Outdoor Scene Train/Test

OST (Outdoor Scenes),OST Training,7 categories images with rich textures

OST300 300 test images of outdoor scences

Download the OST dataset from Google Drive or Baidu Drive.

😆 Image Viewer - HandyViewer

May try HandyViewer - an image viewer that you can switch image with a fixed zoom ratio, easy for comparing image details.