基于注意力的多尺度水下图像增强网络

方明; 刘小晗; 付飞蚺

doi:10.11999/JEIT200836

基于注意力的多尺度水下图像增强网络

doi: 10.11999/JEIT200836

方明^{2, 1},
刘小晗^1, ,,
付飞蚺¹

1.
长春理工大学计算机科学技术学院长春 130022
2.
长春理工大学人工智能学院长春 130022

基金项目: 山东省支持青岛海洋科学与技术试点国家实验室重大科技专项(2018SDKJ0102-6)

详细信息

作者简介:
方明：男，1977年生，副教授，博士，硕士生导师，研究方向为图像处理、计算机视觉

刘小晗：女，1995年生，硕士生，研究方向为图像处理、计算机视觉

付飞蚺：男，1989年生，博士，研究方向为图像处理、计算机视觉

通讯作者:
刘小晗　liuxh928@163.com

中图分类号: TN911.73; TP391
计量
- 文章访问数: 1552
- HTML全文浏览量: 612
- PDF下载量: 211
- 被引次数: 0
出版历程
- 收稿日期: 2020-09-27
- 修回日期: 2021-04-25
- 网络出版日期: 2021-07-14
- 刊出日期: 2021-12-21

Multi-scale Underwater Image Enhancement Network Based on Attention Mechanism

Ming FANG^{2, 1},
Xiaohan LIU^{1
, ,},
Feiran FU¹

1.
School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, China
2.
School of Artificial Intelligence, Changchun University of Science and Technology, Changchun 130022, China

Funds: The Marine S&T Fund of Shandong Province for Pilot National Laboratory for Marine Science and Technology (Qingdao)(2018SDKJ0102-6)

摘要

摘要: 水下图像往往会因为光的吸收和散射而出现颜色退化与细节模糊的现象，进而影响水下视觉任务。该文通过水下成像模型合成更接近水下图像的数据集，以端到端的方式设计了一个基于注意力的多尺度水下图像增强网络。在该网络中引入像素和通道注意力机制，并设计了一个多尺度特征提取模块，在网络开始阶段提取不同层次的特征，通过带跳跃连接的卷积层和注意力模块后得到输出结果。多个数据集上的实验结果表明，该方法在处理合成水下图像和真实水下图像时都能有很好的效果，与现有方法相比能更好地恢复图像颜色和纹理细节。
- 水下图像增强 /
- 深度学习 /
- 注意力机制 /
- 多尺度特征
Abstract: Due to the absorption and scattering, color degradation and detail blurring often occur in underwater images, which will affect the underwater visual tasks. A multi-scale underwater image enhancement network based on attention mechanism is designed in an end-to-end manner by synthesizing dataset closer to underwater images through underwater imaging model. In the network, pixel and channel attention mechanisms are introduced. A new multi-scale feature extraction module is designed to extract the features of different levels at the beginning of the network, and the output results are obtained via a convolution layer and an attention module with skip connections. Experimental results on multiple datasets show that the proposed method is effective in processing both synthetic and real underwater images. It can better recover the color and texture details of images compared with the existing methods.
- Underwater image enhancement /
- Deep learning /
- Attention mechanism /
- Multi-scale features

HTML全文

图 1 水下图像

下载: 全尺寸图片幻灯片

图 2 本文合成的水下图像

下载: 全尺寸图片幻灯片

图 3 网络结构图

下载: 全尺寸图片幻灯片

图 4 平滑空洞卷积

下载: 全尺寸图片幻灯片

图 5 不同方法在合成数据集上的处理效果

下载: 全尺寸图片幻灯片

图 6 不同方法在真实数据集上的处理效果

下载: 全尺寸图片幻灯片

图 7 细节放大图

下载: 全尺寸图片幻灯片

表 1 不同水体中红(R)，绿(G)，蓝(B)通道中参数设置^[23]

水体类型	参数	R	G	B
(1)	${\rm{Nrer}}(\lambda )$	0.79+0.06rand()	0.92+0.06rand()	0.94+0.05rand()
	${B_\lambda }$	0.05+0.15rand()	0.60+0.30rand()	0.70+0.29rand()
(2)	${\rm{Nrer}}(\lambda )$	0.71+0.04rand()	0.82+0.06rand()	0.80+0.07rand()
	${B_\lambda }$	0.05+0.15rand()	0.60+0.30rand()	0.70+0.29rand()
(3)	${\rm{Nrer}}(\lambda )$	0.67	0.73	0.67
	${B_\lambda }$	0.15	0.80	0.75

下载: 导出CSV

表 2 网络参数表

	CINR	MFE	CINR	Maxpool	CINR	Maxpool	CINR	ResBlocks	CINR	Deconv	CINR	Deconv	CINR	CINR	Attn	Conv
卷积核大小	3,1,1		3,1,1	2,2,0	3,1,1	2,2,0	3,1,1	3,1,1	3,1,1	4,2,1	3,1,1	4,2,1	3,1,1	3,1,1		3,1,1
输出通道数	64	64	64	64	128	128	256	256	256	128	128	64	64	64	64	64

下载: 导出CSV

表 3 不同方法在合成数据集上的PSNR和SSIM值

	UDCP	IBLA	ULAP	DUIENet	FUnIE-GAN	本文
PSNR	10.3134	14.8764	14.3421	15.6875	16.7233	28.4583
SSIM	0.5022	0.7225	0.7058	0.7889	0.7601	0.9110

下载: 导出CSV

表 4 图6中图片的不同方法的UIQM值

	图像	UDCP	IBLA	ULAP	DUIENet	FUnIE-GAN	本文
UIQM	第1幅	3.5135	3.8244	3.8666	4.1049	4.3301	4.8344
	第2幅	3.3235	3.5484	3.6567	3.7669	3.3144	3.9077
	第3幅	3.3387	3.2753	3.1166	3.7438	3.9245	4.2782
	第4幅	3.7411	4.1643	3.8395	4.3814	4.4317	4.6122

下载: 导出CSV

表 5 不同方法在真实数据集上的UIQM值

	UDCP	IBLA	ULAP	DUIENet	FUnIE-GAN	本文
UICM	3.8090	3.8621	3.8424	3.5210	4.2954	2.3348
UISM	4.4584	4.7460	4.8385	5.2352	5.2672	5.1937
UIConM	0.7023	0.5915	0.6202	0.6888	0.6779	0.7278
UIQM	3.9349	3.6251	3.7548	4.1079	4.1002	4.2016

下载: 导出CSV

表 6 网络结构消融实验数值结果

	MFE	Attn	感知损失	PSNR	SSIM
无MFE, Attn模块			√	27.8182	0.9057
无Attn模块	√		√	27.8408	0.9092
无感知损失	√	√		28.1272	0.8967
本文完整网络	√	√	√	28.4583	0.9110

下载: 导出CSV

表 7 不同方法运行时间表(s)

	UDCP	IBLA	ULAP	DUIENet	FUnIE-GAN	本文
运行时间	16.81	34.75	2.23	15.30	1.01	15.93

下载: 导出CSV

参考文献(34)

[1]	ZUIDERVELD K. Contrast Limited Adaptive Histogram Equalization[M]. Graphics Gems IV, Amsterdam: Elsevier, 1994: 474–485.
[2]	ANCUTI C, ANCUTI C O, HABER T, et al. Enhancing underwater images and videos by fusion[C]. 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, USA, 2012: 81–88.
[3]	HE Kaiming, SUN Jian, and TANG Xiaoou. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(12): 2341–2353. doi: 10.1109/TPAMI.2010.168
[4]	LI Chongyi, GUO Jichang, CONG Runmin, et al. Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior[J]. IEEE Transactions on Image Processing, 2016, 25(12): 5664–5677. doi: 10.1109/TIP.2016.2612882
[5]	杨爱萍, 郑佳, 王建, 等. 基于颜色失真去除与暗通道先验的水下图像复原[J]. 电子与信息学报, 2015, 37(11): 2541–2547. doi: 10.11999/JEIT150483 YANG Aiping, ZHENG Jia, WANG Jian, et al. Underwater image restoration based on color cast removal and dark channel prior[J]. Journal of Electronics &Information Technology, 2015, 37(11): 2541–2547. doi: 10.11999/JEIT150483
[6]	DREWS P L J, NASCIMENTO E R, BOTELHO S S C, et al. Underwater depth estimation and image restoration based on single images[J]. IEEE Computer Graphics and Applications, 2016, 36(2): 24–35. doi: 10.1109/MCG.2016.26
[7]	REN Wenqi, LIU Si, ZHANG Hua, et al. Single image dehazing via multi-scale convolutional neural networks[C]. The 14th European Conference on Computer Vision, Amsterdam, The Kingdom of the Netherlands, 2016: 154–169.
[8]	AKKAYNAK D and TREIBITZ T. A revised underwater image formation model[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 6723–6732.
[9]	ANWAR S and LI Chongyi. Diving deeper into underwater image enhancement: A survey[EB/OL]. https://arxiv.org/abs/1907.07863, 2019.
[10]	LI Jie, SKINNER K A, EUSTICE R M, et al. WaterGAN: Unsupervised generative network to enable real-time color correction of monocular underwater images[J]. IEEE Robotics and Automation Letters, 2018, 3(1): 387–394. doi: 10.1109/LRA.2017.2730363
[11]	HASHISHO Y, ALBADAWI M, KRAUSE T, et al. Underwater color restoration using u-net denoising autoencoder[EB/OL]. https://arxiv.org/abs/1905.09000. 2019.
[12]	LI Chongyi, GUO Chunle, REN Wenqi, et al. An underwater image enhancement benchmark dataset and beyond[EB/OL]. https://arxiv.org/abs/1901.05495, 2019.
[13]	FABBRI C, ISLAM M J, and SATTAR J. Enhancing underwater imagery using generative adversarial networks[C]. 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 2018: 7159–7165.
[14]	ISLAM M J, XIA Y Y, and SATTAR J. Fast underwater image enhancement for improved visual perception[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 3227–3234. doi: 10.1109/LRA.2020.2974710
[15]	ZHANG Tingting, LI Yujie, and TAKAHASHI S. Underwater image enhancement using improved generative adversarial network[J]. Concurrency and Computation Practice and Experience, 2020: e5841. doi: 10.1002/cpe.5841
[16]	PARK J, HAN D K, and KO H. Adaptive weighted multi-discriminator CycleGAN for underwater image enhancement[J]. Journal of Marine Science and Engineering, 2019, 7(7): 200. doi: 10.3390/jmse7070200
[17]	ANWAR S, LI Chongyi, and PORIKLI F. Deep underwater image enhancement[EB/OL]. https://arxiv.org/abs/1807.03528, 2018.
[18]	LI Chongyi, ANWAR S, and PORIKLI F. Underwater scene prior inspired deep underwater image and video enhancement[J]. Pattern Recognition, 2020, 98: 107038. doi: 10.1016/j.patcog.2019.107038
[19]	SILBERMAN N, HOIEM D, KOHLI P, et al. Indoor segmentation and support inference from RGBD images[C]. The 12th European Conference on Computer Vision, Florence, Italy, 2012: 746–760.
[20]	ZHU Junyan, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]. 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017: 2242–2251.
[21]	CHIANG J Y and CHEN Y C. Underwater image enhancement by wavelength compensation and dehazing[J]. IEEE Transactions on Image Processing, 2012, 21(4): 1756–1769. doi: 10.1109/TIP.2011.2179666
[22]	DING Xueyan, WANG Yafei, YAN Yang, et al. Jointly adversarial network to wavelength compensation and dehazing of underwater images[EB/OL]. https://arxiv.org/abs/1907.05595, 2019.
[23]	ZHOU Yuan and YAN Kangming. Domain adaptive adversarial learning based on physics model feedback for underwater image enhancement[EB/OL]. https://arxiv.org/abs/2002.09315, 2020.
[24]	QIN Xu, WANG Zhilin, BAI Yuanchao, et al. FFA-Net: Feature fusion attention network for single image dehazing[EB/OL]. https://arxiv.org/abs/1911.07559, 2019.
[25]	柳长源, 王琪, 毕晓君, 等. 基于多通道多尺度卷积神经网络的单幅图像去雨方法[J]. 电子与信息学报, 2020, 42(9): 2285–2292. doi: 10.11999/JEIT190755 LIU Changyuan, WANG Qi, BI Xiaojun, et al. Research on rain removal method for single image based on multi-channel and multi-scale CNN[J]. Journal of Electronics &Information Technology, 2020, 42(9): 2285–2292. doi: 10.11999/JEIT190755
[26]	WANG Zhengyang and JI Shuiwang. Smoothed dilated convolutions for improved dense prediction[C]. The 24th ACM SIGKDD International Conference on Knowledge Discovery& Data Mining, London, UK, 2018: 2486–2495.
[27]	CHEN Dongdong, HE Mingming, FAN Qingnan, et al. Gated context aggregation network for image dehazing and deraining[C]. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, USA, 2019: 1375–1383.
[28]	JOHNSON J, ALAHI A, and LI Feifei. Perceptual losses for real-time style transfer and super-resolution[C]. 14th European Conference on Computer Vision, Amsterdam, The Kingdom of the Netherlands, 2016: 694–711.
[29]	SIMONYAN K and ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]. International Conference on Learning Representations (ICLR), San Diego, USA, 2015: 1–5.
[30]	WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. doi: 10.1109/TIP.2003.819861
[31]	PANETTA K, GAO C, and AGAIAN S. Human-visual-system-inspired underwater image quality measures[J]. IEEE Journal of Oceanic Engineering, 2016, 41(3): 541–551. doi: 10.1109/JOE.2015.2469915
[32]	DREWS P JR, DO NASCIMENTO E, MORAES F, et al. Transmission estimation in underwater single images[C]. 2013 IEEE International Conference on Computer Vision Workshops, Sydney, Australia, 2013: 825–830.
[33]	PENG Y T and COSMAN P C. Underwater image restoration based on image blurriness and light absorption[J]. IEEE Transactions on Image Processing, 2017, 26(4): 1579–1594. doi: 10.1109/TIP.2017.2663846
[34]	SONG Wei, WANG Yan, HUANG Dongmei, et al. A rapid scene depth estimation model based on underwater light attenuation prior for underwater image restoration[C]. The 19th Pacific-Rim Conference on Multimedia, Hefei, China, 2018: 678–688.