Skip to content

TensorFlow implementation of "Accurate Image Super-Resolution Using Very Deep Convolutional Network" (CVPR 2016)

Notifications You must be signed in to change notification settings

jangsoopark/VDSR-TensorFlow

Repository files navigation

VDSR-TensorFlow


Introduction

This repository is TensorFlow implementation of VDSR (CVPR16).

You can see more details from paper and author's project page


Network Structure

VDSR-TensorFlow/model/network.py

VDSR Network Structure

  • ILR denotes Interpolated Low Resolution image
  • SR denotes reconstructed super resolution image

Details

  • VDSR structures
Layer (# layers) Filter size Input Dimension Output Dimension Activation Function
Input Layer (1)  3 \times 3 1 64 ReLU
Hidden Layer (18)  3 \times 3 64 64 ReLU
Output Layer (1)  3 \times 3 64 1 -

ReLU (Rectified Linear Unit)

  • I/O
Input (LR) Output (Residual) Reconstructed (LR + Residual)
LR HF SR

Training

Loss Function

_loss_function(self, reg_parameter) in VDSR-TensorFlow/model/___init__.py

  • Basic loss function


Loss(W)=\frac{1}{2}||y-f(x)||^{2}

  • Loss functions for residual learning


Loss(W)=\frac{1}{2}||r-f(x)||^{2}

  • Regularization

    • L2 regularization

    
reg(W)=\frac{\lambda}{2}\sum_{w \in W} {||w||^{2}}

  • Notations

    • W : Weights in VDSR
    • y : ground truth (original high resolution image, HR)
    • x : interpolated low resolution image (ILR)
    • f(x) : reconstructed super resolution image
    • r : residual between HR and ILR
      • r = y-x
    • \lambda : regularization parameter
      • \lambda : 0.0001

Optimization

_optimization_function(self, grad_clip, momentum_rate) in VDSR-TensorFlow/model/___init__.py

  • Optimization Method

    • Stochastic Gradient Descent (SGD) method [Wikipedia]
      • Momentum : 0.9
  • Weight Initialization

  • Learning Rate

    • Extremely high value is used to speed-up convergence rate
    • Initial Learning rate : 0.1
  • Learning Rate Decay

    learning rage in training

    • Learning rate is decreased by factor of 10 for every 20 epochs
  • Adjustable Gradient Clipping

    • Clip individual gradients to [-\frac{\theta}{\gamma}, \frac{\theta}{\gamma}]
      • \theta denotes parameters for gradient clipping
      • \gamma denotes learning rate
  • Epochs : 80


Data Set

Training Data

VDSR-TensorFlow/data/generate_dataset/train_data.m

  • 291 images
  • Bicubic interpolation is used for LR data acquisition
  • Data Augmentations (Rotation, flip) were used
  • Scale Factor : \times 2 \times 3, \times 4
  • Patch size : 41
  • Batch size : 64

Testing Data

VDSR-TensorFlow/data/generate_dataset/test_data.m

  • Set5, Set14, B100, Urban100
  • Bicubic interpolation is used for LR data acquisition
  • Scale Factor : \times 2, \times 3, \times 4

Results

Validation

PSNR performance plot on Set5

  • rotation=[0°, 90°], flip=[left / right], \theta = 0.001
scale2 scale3 scale4
Scale 2 Scale 3 Scale 4
  • rotation=[0°, 90°], flip=[left / right], \theta = 0.0009
scale2 scale3 scale4
Scale 2 Scale 3 Scale 4
  • rotation=[0°, 90°], flip=[left / right], \theta = 0.0008
scale2 scale3 scale4
Scale 2 Scale 3 Scale 4

Objective Quality Assessment

Methods

  • Bicubic Interpolation
    • imresize(..., ..., 'bicubic') in Matlab
  • VDSR (Original)
    • Author's MatConvNet implementation [Code]
  • VDSR (TensorFlow)
    • TensorFlow implementation
    • Train Details for Comparison
      • Gradient Clipping parameter \theta = 0.001
      • Data Augmentation
        • Rotation : 0°, 90°
        • Flip : left / right

Average PSNR/SSIM

PSNR (Peak Signal-to-Noise Ratio)

SSIM (Strutural SIMility)


  • rotation=[0°, 90°], flip=[left / right], \theta = 0.001

  • Set5

scale Bicubic VDSR (Original) VDSR(TensorFlow)
\times 2 33.68 / 0.9304 37.53 / 0.9586 37.07 / 0.9576
\times 3 30.40 / 0.8682 33.66 / 0.9213 33.20 / 0.9171
\times 4 28.43 / 0.8104 31.35 / 0.8838 30.90 / 0.8756
  • Set14
scale Bicubic VDSR (Original) VDSR(TensorFlow)
\times 2 30.24 / 0.8693 33.05 / 0.9214 32.67 / 0.9108
\times 3 27.54 / 0.7746 29.78 / 0.8318 29.58 / 0.8295
\times 4 26.00 / 0.7029 28.02 / 0.7678 27.81 / 0.7627
  • B100
scale Bicubic VDSR (Original) VDSR(TensorFlow)
\times 2 29.56 / 0.8442 31.90 / 0.8960 31.65 / 0.8943
\times 3 27.21 / 0.7401 28.83 / 0.7976 28.66 / 0.7952
\times 4 25.96 / 0.6697 27.29 / 0.7252 27.14 / 0.7217
  • Urban100
scale Bicubic VDSR (Original) VDSR(TensorFlow)
\times 2 26.88 / 0.8410 30.77 / 0.9141 30.20 / 0.9087
\times 3 24.46 / 0.7358 27.14 / 0.8279 26.69 / 0.8178
\times 4 23.14 / 0.6588 25.18 / 0.7525 24.85 / 0.7406

  • rotation=[0°, 90°], flip=[left / right], \theta = 0.0009
  • Set5
scale Bicubic VDSR (Original) VDSR(TensorFlow)
img 33.68 / 0.9304 37.53 / 0.9586 37.14 / 0.9574
img 30.40 / 0.8682 33.66 / 0.9213 33.29 / 0.9171
img 28.43 / 0.8104 31.35 / 0.8838 30.95 / 0.8755
  • Set14
scale Bicubic VDSR (Original) VDSR(TensorFlow)
img 30.24 / 0.8693 33.05 / 0.9214 32.74 / 0.9105
img 27.54 / 0.7746 29.78 / 0.8318 29.62 / 0.8291
img 26.00 / 0.7029 28.02 / 0.7678 27.84 / 0.7628
  • B100
scale Bicubic VDSR (Original) VDSR(TensorFlow)
img 29.56 / 0.8442 31.90 / 0.8960 31.69 / 0.8936
img 27.21 / 0.7401 28.83 / 0.7976 28.68 / 0.7947
img 25.96 / 0.6697 27.29 / 0.7252 27.15 / 0.7214
  • Urban100
scale Bicubic VDSR (Original) VDSR(TensorFlow)
img 26.88 / 0.8410 30.77 / 0.9141 30.29 / 0.9085
img 24.46 / 0.7358 27.14 / 0.8279 26.78 / 0.8186
img 23.14 / 0.6588 25.18 / 0.7525 24.90 / 0.7418

  • rotation=[0°, 90°], flip=[left / right], \theta = 0.0008
  • Set5
scale Bicubic VDSR (Original) VDSR(TensorFlow)
img 33.68 / 0.9304 37.53 / 0.9586 37.08 / 0.9574
img 30.40 / 0.8682 33.66 / 0.9213 33.19 / 0.9166
img 28.43 / 0.8104 31.35 / 0.8838 30.88 / 0.8743
  • Set14
scale Bicubic VDSR (Original) VDSR(TensorFlow)
img 30.24 / 0.8693 33.05 / 0.9214 32.66 / 0.9105
img 27.54 / 0.7746 29.78 / 0.8318 29.54 / 0.8288
img 26.00 / 0.7029 28.02 / 0.7678 27.75 / 0.7617
  • B100
scale Bicubic VDSR (Original) VDSR(TensorFlow)
img 29.56 / 0.8442 31.90 / 0.8960 31.65 / 0.8940
img 27.21 / 0.7401 28.83 / 0.7976 28.66 / 0.7945
img 25.96 / 0.6697 27.29 / 0.7252 27.14 / 0.7209
  • Urban100
scale Bicubic VDSR (Original) VDSR(TensorFlow)
img 26.88 / 0.8410 30.77 / 0.9141 30.13 / 0.9076
img 24.46 / 0.7358 27.14 / 0.8279 26.68 / 0.8171
img 23.14 / 0.6588 25.18 / 0.7525 24.84 / 0.7402

Visual Quality

  • "img002" of Urban100 for scale factor \times 2
  • rotation=[0°, 90°], flip=[left / right], \theta = 0.001
Ground Truth Bicubic VDSR(TensorFlow)
img002_gt img002_lr img002_sr
img002_gt img002_lr img002_sr
  • rotation=[0°, 90°], flip=[left / right], \theta = 0.0009
Ground Truth Bicubic VDSR(TensorFlow)
img002_gt img002_lr img002_sr
img002_gt img_002_lr img
  • rotation=[0°, 90°], flip=[left / right], \theta = 0.0008
Ground Truth Bicubic VDSR(TensorFlow)
img002_gt img002_lr img002_sr
img002_gt img_002_lr img

Usage

On Windows

  • run.bat

On Linux

  • run.sh

Prerequisites

  • Python 3

    pip install -r requirements.txt

    • tensorflow
    • h5py
    • numpy + mkl
    • scipy
    • Pillow
    • skimage
  • Matlab

    • For data generation and performance evaluation
    • image processing toolbox

Training Command

  • in run.bat/sh
    • python main.py --model_name=vdsr --is_train=True --grad_clip=1e-3

If you want to change other parameters for training, please see the file

VDSR-TensorFlow/model/configurations.py

Testing Command

in run.bat

python main.py --model_name=vdsr --is_train=False

Trained checkpoint in experiments

Training dataset

  • vdsr_train.h5
    • rotation=[0°, 90°], flip=[left / right] [download]

math equations are converted with upmath.me