Super-Resolution Inpainting of Low-resolution Randomly Occluded Face Images

REN Kun; LI Zhengzhen; GUI Yuanze; FAN Chunqi; LUAN Heng

doi:10.11999/JEIT231262

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2022 >

REN Kun, LI Zhengzhen, GUI Yuanze, FAN Chunqi, LUAN Heng. Super-Resolution Inpainting of Low-resolution Randomly Occluded Face Images[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT231262

Citation:

REN Kun, LI Zhengzhen, GUI Yuanze, FAN Chunqi, LUAN Heng. Super-Resolution Inpainting of Low-resolution Randomly Occluded Face Images[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT231262

Citation:

PDF( 4805 KB)

Super-Resolution Inpainting of Low-resolution Randomly Occluded Face Images

doi: 10.11999/JEIT231262

REN Kun^{1, 2, 3
,
,},
LI Zhengzhen^{1, 2, 3},
GUI Yuanze¹,
FAN Chunqi¹,
LUAN Heng⁴

1.
Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
2.
Engineering Research Center of Digital Community, Ministry of Education, Beijing University of Technology, Beijing 100124, China
3.
Beijing Laboratory for Urban Mass Transit, Beijing University of Technology, Beijing 100124, China
4.
TravelSky Technology Limited, Beijing 101300, China

Funds: National Key Research and Development Project (2023YFC3904605)

Received Date: 2023-11-15
Rev Recd Date: 2024-02-05

Available Online: 2024-03-04

Abstract

Abstract

An end-to-end quadruple Super-Resolution Inpainting Generative Adversarial Network (SRIGAN) is proposed in this paper, for low-resolution random occlusion face images. The generative network consists of an encoder, a feature compensation subnetwork, and a decoder constructed with a pyramid attention module. The discriminant network is an improved Patch discriminant network. The network can effectively learn the absent features of the occluded region through a feature compensation subnetwork and a two-stage training strategy. Then, the information is constructed with the decoder with a pyramid attention module and multi-scale reconstruction loss. Hence, the generative network can transform a low-resolution occlusion image into a quadruple high-resolution complete image. Furthermore, the improvements of the loss function and Patch discriminant network are employed to ensure the stability of network training and enhance the performance of the generated network. The effectiveness of the proposed algorithm is verified by comparison and module verification experiments.
- Image inpainting,
- Super-Resolution(SR),
- Generative Adversarial Network(GAN),
- Pyramid attention

FullText(HTML)

References(25)

References

[1]	刘颖, 张艺轩, 佘建初, 等. 人脸去遮挡新技术研究综述[J]. 计算机科学与探索, 2021, 15(10): 1773–1794. doi: 10.3778/j.issn.1673-9418.2103092. LIU Ying, ZHANG Yixuan, SHE Jianchu, et al. Review of new face occlusion inpainting technology research[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(10): 1773–1794. doi: 10.3778/j.issn.1673-9418.2103092.
[2]	卢启萌, 毛晓, 凌嵘, 等. 口罩佩戴对人像鉴定的影响[J]. 中国司法鉴定, 2021(5): 89–94. doi: 10.3969/j.issn.1671-2072.2021.05.010. LU Qimeng, MAO Xiao, LING Rong, et al. Influence of mask wearing on identification of human images[J]. Chinese Journal of Forensic Sciences, 2021(5): 89–94. doi: 10.3969/j.issn.1671-2072.2021.05.010.
[3]	廖海斌, 陈友斌, 陈庆虎. 基于非局部相似字典学习的人脸超分辨率与识别[J]. 武汉大学学报:信息科学版, 2016, 41(10): 1414–1420. doi: 10.13203/j.whugis20140498. LIAO Haibin, CHEN Youbin, and CHEN Qinghu. Non-local similarity dictionary learning based super-resolution for improved face recognition[J]. Geomatics and Information Science of Wuhan University, 2016, 41(10): 1414–1420. doi: 10.13203/j.whugis20140498.
[4]	王山豹, 梁栋, 沈玲. 利用多模态注意力机制生成网络的图像修复[J]. 计算机辅助设计与图形学学报, 2023, 35(7): 1109–1121. doi: 10.3724/SP.J.1089.2023.19578. WANG Shanbao, LIANG Dong, and SHEN Ling. Image inpainting with multi-modal attention mechanism generative networks[J]. Journal of Computer-Aided Design & Computer Graphics, 2023, 35(7): 1109–1121. doi: 10.3724/SP.J.1089.2023.19578.
[5]	张子迎, 周华. 强化结构的数字壁画病害修复算法研究[J]. 系统仿真学报, 2022, 34(7): 1524–1531. doi: 10.16182/j.issn1004731x. joss.21-0034. ZHANG Ziying and ZHOU Hua. Research on inpainting algorithm of digital murals based on enhanced structural information[J]. Journal of System Simulation, 2022, 34(7): 1524–1531. doi: 10.16182/j.issn1004731x.joss.21-0034.
[6]	BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: A randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 24. doi: 10.1145/1531326.1531330.
[7]	BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]. The 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, USA, 2000: 417–424. doi: 10.1145/344779.344972.
[8]	ESEDOGLU S and SHEN Jianhong. Digital inpainting based on the Mumford–Shah–Euler image model[J]. European Journal of Applied Mathematics, 2002, 13(4): 353–370. doi: 10.1017/S0956792502004904.
[9]	YU Jiahui, LIN Zhe, YANG Jimei, et al. Generative image inpainting with contextual attention[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 5505–5514. doi: 10.1109/CVPR.2018.00577.
[10]	NAZERI K, NG E, JOSEPH T, et al. EdgeConnect: Generative image inpainting with adversarial edge learning[J]. arXiv preprint arXiv: 1901.00212, 2019.
[11]	LI Wenbo, LIN Zhe, ZHOU Kun, et al. MAT: Mask-aware transformer for large hole image inpainting[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 10758–10768. doi: 10.1109/CVPR52688.2022.01049.
[12]	ZENG Yanhong, FU Jianlong, CHAO Hongyang, et al. Aggregated contextual transformations for high-resolution image inpainting[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(7): 3266–3280. doi: 10.1109/TVCG.2022.3156949.
[13]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]. The 27th International Conference on Neural Information Processing Systems, Montréal, Canada, 2014: 1384–1393.
[14]	GUO Yong, CHEN Jian, WANG Jingdong, et al. Closed-loop matters: Dual regression networks for single image super-resolution[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 5407–5416. doi: 10.1109/CVPR42600.2020.00545.
[15]	MEI Yiqun, FAN Yuchen, ZHANG Yulun, et al. Pyramid attention network for image restoration[J]. International Journal of Computer Vision, 2023, 131(12): 3207–3225. doi: 10.1007/s11263-023-01843-5.
[16]	汪荣贵, 雷辉, 杨娟, 等. 基于自相似特征增强网络结构的图像超分辨率重建[J]. 光电工程, 2022, 49(5): 210382. doi: 10.12086/oee.2022.210382. WANG Ronggui, LEI Hui, YANG Juan, et al. Self-similarity enhancement network for image super-resolution[J]. Opto-Electronic Engineering, 2022, 49(5): 210382. doi: 10.12086/oee.2022.210382.
[17]	黄友文, 唐欣, 周斌. 结合双注意力和结构相似度量的图像超分辨率重建网络[J]. 液晶与显示, 2022, 37(3): 367–375. doi: 10.37188/CJLCD.2021-0178. HUANG Youwen, TANG Xin, and ZHOU Bin. Image super-resolution reconstruction network with dual attention and structural similarity measure[J]. Chinese Journal of Liquid Crystals and Displays, 2022, 37(3): 367–375. doi: 10.37188/CJLCD.2021-0178.
[18]	CHEN Xiangyu, WANG Xintao, ZHOU Jiantao, et al. Activating more pixels in image super-resolution transformer[C]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Canada, 2023: 22367–22377. doi: 10.1109/CVPR52729.2023.02142.
[19]	LIANG Jingyun, CAO Jiezhang, SUN Guolei, et al. SwinIR: Image restoration using swin transformer[C]. 2021 IEEE/CVF International Conference on Computer Vision Workshops, Montreal, Canada, 2021: 1833–1844. doi: 10.1109/ICCVW54120.2021.00210.
[20]	ARJOVSKY M, CHINTALA S, and BOTTOU L. Wasserstein GAN[J]. arXiv preprint arXiv: 1701.07875, 2017.
[21]	GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of wasserstein GANs[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 5767–5777.
[22]	MIYATO T, KATAOKA T, KOYAMA M, et al. Spectral normalization for generative adversarial networks[J]. arXiv preprint arXiv: 1802.05957, 2018.
[23]	ISOLA P, ZHU Junyan, ZHOU Tinghui, et al. Image-to-image translation with conditional adversarial networks[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 1125–1134. doi: 10.1109/CVPR.2017.632.
[24]	JOHNSON J, ALAHI A, and FEI-FEI L. Perceptual losses for real-time style transfer and super-resolution[C]. The 14th European Conference on Computer Vision, Amsterdam, the Netherlands, 2016: 694–711. doi: 10.1007/978-3-319-46475-6_43.
[25]	LIU Guilin, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C]. The 15th European Conference on Computer Vision, Munich, Germany, 2018: 85–100. doi: 10.1007/978-3-030-01252-6_6.