多级跳线连接的深度残差网络超分辨率重建

赵小强; 宋昭漾

doi:10.11999/JEIT190036

多级跳线连接的深度残差网络超分辨率重建

doi: 10.11999/JEIT190036

赵小强^{1, 2, 3, ,},
宋昭漾¹

1.
兰州理工大学电气工程与信息工程学院兰州 730050
2.
甘肃省工业过程先进控制重点实验室兰州 730050
3.
兰州理工大学国家级电气与控制工程实验教学中心兰州 730050

基金项目: 国家科学自然基金(61763029, 61873116)

详细信息

作者简介:
赵小强：男，1969年生，博士生导师，教授，主要研究方向为故障诊断，图像处理，生产调度等

宋昭漾：男，1995年生，硕士生，研究方向为图像处理

通讯作者:
赵小强　xqzhao@lut.cn

中图分类号: TP391
计量
- 文章访问数: 3058
- HTML全文浏览量: 1219
- PDF下载量: 112
- 被引次数: 0
出版历程
- 收稿日期: 2019-01-15
- 修回日期: 2019-06-30
- 网络出版日期: 2019-07-19
- 刊出日期: 2019-10-01

Super-Resolution Reconstruction of Deep Residual Network with Multi-Level Skip Connections

Xiaoqiang ZHAO^{1, 2, 3
, ,},
Zhaoyang SONG¹

1.
College of Electrical Engineering and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China
2.
Key Laboratory of Gansu Advanced Control for Industrial Processes, Lanzhou University of Technology, Lanzhou 730050, China
3.
National Experimental Teaching Center of Electrical and Control Engineering, Lanzhou University of Technology, Lanzhou 730050, China

Funds: The National Natural Science Foundation of China (61763029, 61873116)

摘要

摘要: 由于快速的卷积神经网络超分辨率重建算法(FSRCNN)卷积层数少、相邻卷积层的特征信息之间缺乏关联性，因此难以提取到图像深层信息导致图像超分辨率重建效果不佳。针对此问题，该文提出多级跳线连接的深度残差网络超分辨率重建方法。首先，该方法设计了多级跳线连接的残差块，在多级跳线连接的残差块基础上构造了多级跳线连接的深度残差网络，解决相邻卷积层的特性信息缺乏关联性的问题；然后，使用随机梯度下降法(SGD)以可调节的学习率策略对多级跳线连接的深度残差网络进行训练，得到该网络超分辨率重建模型；最后，将低分辨率图像输入到多级跳线连接的深度残差网络超分辨率重建模型中，通过多级跳线连接的残差块得到预测的残差特征值，再将残差图像和低分辨率图像组合在一起转化为高分辨率图像。该文方法与bicubic, A+, SRCNN, FSRCNN和ESPCN算法在Set5和Set14测试集上进行了对比测试，在视觉效果和评价指标数值上该方法都优于其它对比算法。
- 超分辨率重建 /
- 深度残差网络 /
- 多级跳线连接的残差块 /
- 随机梯度下降法
Abstract: The Fast Super-Resolution Convolutional Neural Network algorithm (FSRCNN) is difficult to extract deep image information due to the small number of convolution layers and the correlation lack between the feature information of adjacent convolutional layers. To solve this problem, a deep residual network super-resolution reconstruction method with multi-level skip connections is proposed. Firstly, a residual block with multi-level skip connections is designed to solve the problem that the characteristic information of adjacent convolutional layers lacks relevance. A deep residual network with multi-level skip connections is constructed on the basis of the residual block. Then, the deep residual network connected to the multi-level skip is trained by using the adaptive gradient rate strategy of Stochastic Gradient Descent (SGD) method and the network super-resolution reconstruction model is obtained. Finally, the low-resolution image is input into the deep residual network super-resolution reconstruction model with the multi-level skip connections, and the residual eigenvalue is obtained by the residual block connected the multi-level skip connections. The residual eigenvalue and the low resolution image are combined and converted into a high resolution image. The proposed method is compared with the bicubic, A+, SRCNN, FSRCNN and ESPCN algorithms in the Set5 and Set14 test sets. The proposed method is superior to other comparison algorithms in terms of visual effects and evaluation index values.
- Super-resolution reconstruction /
- Deep residual network /
- Residual network with multi-level skip connections /
- Stochastic Gradient Descent (SGD)

HTML全文

图 1 残差块结构图

下载: 全尺寸图片幻灯片

图 2 多级跳线连接的残差块结构图

下载: 全尺寸图片幻灯片

图 3 相邻两个多级跳线连接的残差块结构图

下载: 全尺寸图片幻灯片

图 4 多级跳线连接的深度残差网络结构图

下载: 全尺寸图片幻灯片

图 5 不同跳线系数测得的峰值信噪比(PSNR)曲线

下载: 全尺寸图片幻灯片

图 6 Set5 测试集中的baby_GT重建对比图

下载: 全尺寸图片幻灯片

表 1 在Set5测试集上的测得的PSNR(dB)/SSIM值

放大因子	Bicubic^[27]	A+^[28]	SRCNN^[18]	FSRCNN^[19]	ESPCN^[21]	本文方法
2	33.66/0.9299	36.54/0.9544	36.66/0.9542	37.00/0.9558	37.06/0.9559	37.35/0.9573
3	30.39/0.8682	32.58/0.9088	32.75/0.9090	33.16/0.9104	33.13/0.9135	33.45/0.9162
4	28.42/0.8104	30.28/0.8603	30.48/0.8628	30.71/0.8657	30.90/0.8673	31.07/0.8751

下载: 导出CSV

表 2 在Set14测试集上的测得的PSNR(dB)/ SSIM值

放大因子	Bicubic	A+	SRCNN	FSRCNN	ESPCN	本文方法
2	30.24/0.8688	32.28/0.9056	32.42/0.9063	32.63/0.9088	32.75/0.9098	33.34/0.9143
3	27.55/0.7742	29.13/0.8188	29.28/0.8209	29.43/0.8242	29.49/0.8271	30.09/0.8512
4	26.00/0.7027	27.32/0.7491	27.49/0.7503	27.59/0.7535	27.73/0.7637	28.26/0.7893

下载: 导出CSV

参考文献(29)

THORNTON M W, ATKINSON P M, and HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3): 473–491. doi: 10.1080/01431160500207088

PELED S and YESHURUN Y. Superresolution in MRI: Application to human white matter fiber tract visualization by diffusion tensor imaging[J]. Magnetic Resonance in Medicine, 2001, 45(1): 29–35. doi: 10.1002/1522-2594(200101)45

ZOU W W W and YUEN P C. Very low resolution face recognition problem[J]. IEEE Transactions on Image Processing, 2012, 21(1): 327–340. doi: 10.1109/TIP.2011.2162423

LU Huimin, LI Yujie, CHEN Min, et al. Brain intelligence: Go beyond artificial intelligence[J]. Mobile Networks and Applications, 2018, 23(2): 368–375. doi: 10.1007/s11036-017-0932-8

KOCH M. Artificial intelligence is becoming natural[J]. Cell, 2018, 173(3): 531–533. doi: 10.1016/j.cell.2018.04.007

LEO M, MEDIONI G, TRIVEDI M, et al. Computer vision for assistive technologies[J]. Computer Vision and Image Understanding, 2017, 154: 1–15. doi: 10.1016/j.cviu.2016.09.001

ZHU Hong, TANG Xinming, XIE Junfeng, et al. Spatio-temporal super-resolution reconstruction of remote-sensing images based on adaptive multi-scale detail enhancement[J]. Sensors, 2018, 18(2): 498. doi: 10.3390/s18020498

SHI Jun, LIU Qingping, WANG Chaofeng, et al. Super-resolution reconstruction of MR image with a novel residual learning network algorithm[J]. Physics in Medicine & Biology, 2018, 63(8): 085011. doi: 10.1088/1361-6560/aab9e9

SU Heng, ZHOU Jie, and ZHANG Zhihao. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202–1213. doi: 10.3724/SP.J.1004.2013.01202

GRIBBON K T and BAILEY D G. A novel approach to real-time bilinear interpolation[C]. The 2nd IEEE International Workshop on Electronic Design, Test and Applications, Perth, Australia, 2004: 126–131. doi: 10.1109/DELTA.2004.10055.

FRITSCH F N and CARLSON R E. Monotone piecewise cubic interpolation[J]. SIAM Journal on Numerical Analysis, 1980, 17(2): 238–246. doi: 10.1137/0717021

STARK H and OSKOUI P. High-resolution image recovery from image-plane arrays, using convex projections[J]. Journal of the Optical Society of America A, 1989, 6(11): 1715–1726. doi: 10.1364/JOSAA.6.001715

PATANAVIJIT V and JITAPUNKUL S. An iterative super-resolution reconstruction of image sequences using fast affine block-based registration with BTV regularization[C]. Proceedings of 2006 IEEE Asia Pacific Conference on Circuits and Systems, Singapore, 2006: 1717–1720. doi: 10.1109/APCCAS.2006.342128.

ZHOU Fei, YANG Wenming, and LIAO Qingmin. Interpolation-based image super-resolution using multisurface fitting[J]. IEEE Transactions on Image Processing, 2012, 21(7): 3312–3318. doi: 10.1109/TIP.2012.2189576

LIN Zhouchen and SHUM H Y. Fundamental limits of reconstruction-based superresolution algorithms under local translation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 83–97. doi: 10.1109/TPAMI.2004.1261081

YOUNG T, HAZARIKA D, PORIA S, et al. Recent trends in deep learning based natural language processing[J]. IEEE Computational Intelligence Magazine, 2018, 13(3): 55–75. doi: 10.1109/MCI.2018.2840738

KERMANY D S, GOLDBAUM M, CAI Wenjia, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning[J]. Cell, 2018, 172(5): 1122–1131.e9. doi: 10.1016/j.cell.2018.02.010

DONG Chao, LOY C C, HE Kaiming, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307. doi: 10.1109/TPAMI.2015.2439281

DONG Chao, LOY C C, and TANG Xiaoou. Accelerating the super-resolution convolutional neural network[C]. The 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 391–407. doi: 10.1007/978-3-319-46475-6_25.

KIM J, LEE J K, and LEE K M. Deeply-recursive convolutional network for image super-resolution[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1637–1645. doi: 10.1109/CVPR.2016.181.

SHI Wenzhe, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1874–1883. doi: 10.1109/CVPR.2016.207.

LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 4681–4690. doi: 10.1109/CVPR.2017.19.

HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.

KIM J, LEE J K, and LEE K M. Accurate image super-resolution using very deep convolutional networks[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1646–1654. doi: 10.1109/CVPR.2016.182.

YUAN Fei, HUANG Lianfen, and YAO Yan. An improved PSNR algorithm for objective video quality evaluation[C]. 2007 Chinese Control Conference, Hunan, China, 2007: 376–380. doi: 10.1109/CHICC.2006.4347144.

WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. doi: 10.1109/TIP.2003.819861

GAO Shengkui and GRUEV V. Bilinear and bicubic interpolation methods for division of focal plane polarimeters[J]. Optics Express, 2011, 19(27): 26161–26173. doi: 10.1364/OE.19.026161

TIMOFTE R, DE SMET V, and VAN GOOL L. A+: Adjusted anchored neighborhood regression for fast super-resolution[C]. The 12th Asian Conference on Computer Vision, Singapore, 2014: 111–126. doi: 10.1007/978-3-319-16817-3_8.

ZEYDE R, ELAD M, and PROTTER M. On single image scale-up using sparse-representations[C]. The 7th International Conference on Curves and Surfaces, Avignon, France, 2010: 711–730. doi: 10.1007/978-3-642-27413-8_47.

施引文献

资源附件(0)

访问统计