SeaNet-PyTorch

This repository is an official PyTorch implementation of the paper "Soft-edge Assisted Network for Single Image Super-Resolution". (TIP 2020)

Paper can be download from SeaNet

Homepage: SeaNet

The task of single image super-resolution (SISR) is a highly ill-posed inverse problem since reconstructing the highfrequency details from a low-resolution image is challenging. Most previous CNN-based super-resolution (SR) methods tend to directly learn the mapping from the low-resolution image to the high-resolution image through some complex convolutional neural networks. However, the method of blindly increasing the depth of the network is not the best choice because the performance improvement of such methods is marginal but the computational cost is huge. A more efficient method is to integrate the image prior knowledge into the model to assist the image reconstruction. Indeed, the soft-edge has been widely applied in many computer vision tasks as the role of an important image feature. In this paper, we propose a Soft-edge assisted Network (SeaNet) to reconstruct the high-quality SR image with the help of image soft-edge. The proposed SeaNet consists of three sub-nets: a rough image reconstruction network (RIRN), a soft-edge reconstruction network (Edge-Net), and an image refinement network (IRN). The complete reconstruction process consists of two stages. In Stage-I, the rough SR feature maps and the SR soft-edge are reconstructed by the RIRN and Edge-Net, respectively. In Stage-II, the outputs of the previous stages are fused and then fed to the IRN for high-quality SR image reconstruction. Extensive experiments show that our SeaNet converges rapidly and achieves excellent performance under the assistance of image soft-edge.

###SEAN is another name for SeaNet, which is convenient for us to conduct experiments.

All reconstructed SR images can be download from SR_Images

All test datasets (Preprocessed HR images) can be downloaded from here.

All original test datasets (HR images) can be downloaded from here.

Prerequisites:

1. Python 3.6

2. PyTorch >= 0.4.0

3. numpy

4. skimage

5. imageio

6. matplotlib

7. tqdm

For more informaiton, please refer to EDSR and RCAN.

Document

Train/ : all train files are stored here

Test/ : all test files are stored here

README.md : read me first

demo.sh : all running instructions

Dataset

We use DIV2K dataset to train our model. Please download it from here or SNU_CVLab.

Extract the file and put it into the Train/dataset.

Only DIV2K is used as the training dataset, and Flickr2K is not used as the training dataset !!!

We use generate_edge.m to extract the soft-edge of DIV2K and put them into the Train/dataset/DIV2K/DIV2K_train_EDGE

##Training

Using --ext sep_reset argument on your first running.

You can skip the decoding part and use saved binaries with --ext sep argument in second time.

cd Train/

# SEAN x2  LR: 48 * 48  HR: 96 * 96

python main.py --template SEAN --save SEAN_X2 --scale 2 --reset --save_results --patch_size 96 --ext sep_reset

# SEAN x3  LR: 48 * 48  HR: 144 * 144

python main.py --template SEAN --save SEAN_X3 --scale 3 --reset --save_results --patch_size 144 --ext sep_reset

# SEAN x4  LR: 48 * 48  HR: 192 * 192

python main.py --template SEAN --save SEAN_X4 --scale 4 --reset --save_results --patch_size 192 --ext sep_reset

##Testing

All original test datasets (HR images) can be downloaded from here.

Different from previous works to select the best weight as the final model weight, we use the weight of the last epoch as our final model weight directly.

Using pre-trained model for test, all test datasets must be pretreatment by Prepare_TestData_HR_LR.m and all pre-trained model should be put into Test/model/ first.

#SEAN x2
python main.py --data_test MyImage --scale 2 --model SEAN --pre_train ../model/SEAN_x2.pt --test_only --save_results --chop --save "SEAN" --testpath ../LR/LRBI --testset Set5

#SEAN+ x2
python main.py --data_test MyImage --scale 2 --model SEAN --pre_train ../model/SEAN_x2.pt --test_only --save_results --chop --self_ensemble --save "SEAN_plus" --testpath ../LR/LRBI --testset Set5

#SEAN x3
python main.py --data_test MyImage --scale 3 --model SEAN --pre_train ../model/SEAN_x3.pt --test_only --save_results --chop --save "SEAN" --testpath ../LR/LRBI --testset Set5

#SEAN+ x3
python main.py --data_test MyImage --scale 3 --model SEAN --pre_train ../model/SEAN_x3.pt --test_only --save_results --chop --self_ensemble --save "SEAN_plus" --testpath ../LR/LRBI --testset Set5

#SEAN x4
python main.py --data_test MyImage --scale 4 --model SEAN --pre_train ../model/SEAN_x4.pt --test_only --save_results --chop --save "SEAN" --testpath ../LR/LRBI --testset Set5

#SEAN+ x4
python main.py --data_test MyImage --scale 4 --model SEAN --pre_train ../model/SEAN_x4.pt --test_only --save_results --chop --self_ensemble --save "SEAN_plus" --testpath ../LR/LRBI --testset Set5

We also introduce self-ensemble strategy to improve our SEAN and denote the self-ensembled version as SEAN+.

More running instructions can be found in demo.sh.

Performance

We use Test/PSNR_SSIM_Results_BI_model.txt for PSRN/SSIM test.

Training curves:

This work was completed in 2018, a long time ago, so there may be omissions in the code finishing process. If you have any questions, please contact me!

@InProceedings{fang2020soft,
    title = {Soft-Edge Assisted Network for Single Image Super-Resolutionn},
    author = {Fang, Faming and Li, Juncheng and Zeng, Tieyong},
    booktitle = {IEEE Transactions on Image Processing},
    volume = {29},
    pages = {4656--4668},
    year = {2020},
    publisher = {IEEE}
}

@InProceedings{fang2020multilevel,
    title = {Multi-level Edge Features Guided Network for Image Denoising},
    author = {Fang, Faming and Li, Juncheng, Yuan Yiting, Zeng, Tieyong, and Zhang Guxiu},
    booktitle = {IEEE Transactions on Neural Networks and Learning Systems},
    publisher = {IEEE}
}

This implementation is for non-commercial research use only. If you find this code useful in your research, please cite the above papers.