An Eyeglasses Removal Method for Fine-grained Face Recognition
-
摘要: 为解决眼镜遮挡会降低人脸识别性能的难点,借鉴深度卷积神经网络在超分辨率方面的成功应用,该文提出一种用于细粒度人脸识别的眼镜自动去除方法ERCNN。用卷积层、池化层、MFM特征选取模块和反卷积层设计ERCNN网络模型,自动学习戴眼镜和未戴眼镜人脸图像对之间的映射关系,实现端到端的眼镜去除。然后,收集大量监控场景下的人脸图像,以及互联网上公开的人脸图像作为训练集;同时构建SLLFW数据集,作为眼镜去除和人脸识别的测试集。最后,通过与传统的眼镜去除方法进行对比试验,该文算法的各项评价指标优于传统方法,能有效的去除真实人脸图像中眼镜;同时在SLLFW人脸数据集上形成的全框眼镜、半框眼镜和无框眼镜人脸数据集上对多种人脸识别算法进行对比试验。试验表明,在FAR为1%的情况下,利用该文方法对F-SLLFW, H-SLLFW和R-SLLFW数据集的人脸图像进行眼镜去除后,SphereFace算法的TAR分别达到90.05%, 91.14%和92.33%,比未去除眼镜的识别率分别提高了3.92%, 3.08%和1.26%;同样,在FAR为0.1%的情况下,比SphereFace算法的TAR分别提高了10.06%, 4.29%和2.13%,说明该文方法有助于提升细粒度人脸识别的识别精度。Abstract: In order to solve the problem that eyeglasses reduce often the performance of face recognition, based on the successful application of deep convolution neural network in super-resolution, This paper proposes an automatic eyeglasses removal method ERCNN (Eyeglasses Removal CNN) for fine-grained face recognition. Specifically, the ERCNN network which is designed based on the convolution layer, pool layer, MFM (Max Feature Map)feature selection module and deconvolution layer, are automatically learned the mapping relationship between facial images with eyeglasses and their counterparts without eyeglasses to realize end-to-end eyeglasses removal. Then, massive facial images are captured through surveillance equipment and collected from the Internet as the training set. And, SLLFW data set is established, which is used as the test set of eyeglasses removal and face recognition. The experiment show that the proposed method can better effectively remove the eyeglasses from the real facial image than the traditional eyeglasses removal methods, and the evaluation index of the method is better than other methods. In addition, several face recognition methods are tested separately on the facial images formed by SLLFW data set. Experiments show that when the FAR (False Accept Rate) is 1%, the TAR (True Accept Rate) of the Sphereface method reaches 90.05%, 91.14% and 92.33%, which is 3.92%, 3.08% and 1.26% higher than the Sphereface method is not used to remove the eyeglasses from the F-SLLFW, H-SLLF and R-SLLFW, respectively. Similarly, when the FAR 0.1%, the TAR of Sphereface method is increased by 10.06%, 3.08% and 1.26% respectively. Therefore, the proposed method can better improve the recognition accuracy of fine-grained face recognition.
-
表 1 不同方法以及不同类型的眼镜去除性能对比
人脸对齐 眼镜类型 方法 评价指标 PSNR (dB) SSIM MSE IFC 128×128 全框 ERCNN 34.7 0.98 22.03 7.0 PCA 24.2 0.85 247.22 6.1 半框 ERCNN 35.7 0.99 17.5 7.3 PCA 25.5 0.87 183.27 6.2 无框 ERCNN 35.7 0.99 17.5 7.5 PCA 26.0 0.88 163.34 6.3 平均值 ERCNN 35.3 0.98 19.01 7.2 PCA 25.2 0.86 197.94 6.2 112×96 全框 ERCNN 35.6 0.99 17.91 7.3 PCA 24.3 0.84 241.59 4.5 半框 ERCNN 36.3 0.99 15.24 7.6 PCA 25.0 0.85 205.63 4.5 无框 ERCNN 36.1 0.99 15.96 7.7 PCA 24.9 0.85 210.42 4.6 平均值 ERCNN 36.0 0.99 16.37 7.5 PCA 24.7 0.84 219.21 4.5 表 2 不同人脸识别的识别性能对比(%)
数据集 FAR TAR ① ② ③ ④ LFW 1 99.03 98.13 99.07 99.40 0.1 95.50 91.27 95.90 97.04 SLLFW 1 93.17 78.43 92.53 96.11 0.1 84.33 62.23 85.43 91.38 F-SLLFW 1 80.87 55.77 83.20 86.13 0.1 68.27 39.87 70.70 71.73 本文方法对F-SLLFW进行处理后 1 86.47 64.47 88.63 90.05 0.1 77.07 48.10 77.87 81.79 H-SLLFW 1 83.10 60.70 86.27 88.06 0.1 72.80 43.53 75.10 78.36 本文方法对H-SLLFW进行处理后 1 87.60 67.07 89.93 91.14 0.1 80.70 45.53 78.83 82.65 R-SLLFW 1 87.07 66.10 89.37 91.70 0.1 79.53 49.67 76.73 82.76 本文方法对R-SLLFW进行处理后 1 88.53 67.50 89.13 92.33 0.1 81.07 47.80 77.43 84.89 -
[1] 李小薪, 梁荣华. 有遮挡人脸识别综述: 从子空间回归到深度学习[J]. 计算机学报, 2018, 41(1): 177–207. doi: 10.11897/SP.J.1016.2018.00177LI Xiaoxin and LIANG Ronghua. A review for face recognition with occlusion: From subspace regression to deep learning[J]. Chinese Journal of Computers, 2018, 41(1): 177–207. doi: 10.11897/SP.J.1016.2018.00177 [2] WU Chenyu, LIU Ce, SHUM H Y, et al. Automatic eyeglasses removal from face images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(3): 322–336. doi: 10.1109/TPAMI.2004.1262319 [3] DU Cheng and SU Guangda. Eyeglasses removal from facial images[J]. Pattern Recognition Letters, 2005, 26(14): 2215–2220. doi: 10.1016/j.patrec.2005.04.002 [4] PARK J S, OH Y H, AHN S C, et al. Glasses removal from facial image using recursive error compensation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(5): 805–811. doi: 10.1109/TPAMI.2005.103 [5] HEO J and SAVVIDES M. Face pose correction with eyeglasses and occlusions removal[C]. 2007 IEEE Conference Biometrics Symposium, Baltimore, USA, 2007: 1–6. doi: 10.1109/BCC.2007.4430551. [6] JIA Xiaodong and GUO Jiangling. Eyeglasses removal from facial image based on phase congruency[C]. The 3rd IEEE International Congress on Image and Signal Processing, Yantai, China, 2010: 1859–1862, doi: 10.1109/CISP.2010.5647366. [7] WANG Yuankai, JANG J H, TSAI L W, et al. Improvement of face recognition by eyeglass removal[C]. The 6th IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Darmstadt, Germany, 2010: 228–231. doi: 10.1109/IIHMSP.2010.64. [8] CHENG Wenchang, LIAO H C, PAN M H, et al. A fatigue detection system with eyeglasses removal[C]. The 15th IEEE International Conference on Advanced Communications Technology, PyeongChang, South Korea, 2013: 331–335. [9] 李根, 李文辉. 基于思维进化的机器学习的遮挡人脸识别[J]. 吉林大学学报: 工学版, 2014, 44(5): 1410–1416.LI Gen and LI Wenhui. Face occlusion recognition based on MEBML[J]. Journal of Jilin University:Engineering and Technology Edition, 2014, 44(5): 1410–1416. [10] GUO Pei and SU Fei. Enhanced PCA reconstruction method for eyeglass frame auto-removal[C]. The 4th IEEE International Conference on Network Infrastructure and Digital Content, Beijing, China, 2014: 359–363. doi: 10.1109/ICNIDC.2014.7000325. [11] 吴从中, 刘渠芬, 詹曙. 基于稀疏表达的人脸遮挡物去除[J]. 合肥工业大学学报: 自然科学版, 2015, 38(5): 615–617, 680.WU Congzhong, LIU Qufen, and ZHAN Shu. A face occlusion removal method based on sparse representation[J]. Journal of Hefei University of Technology, 2015, 38(5): 615–617, 680. [12] FERNÁNDEZ A, GARCÍA R, USAMENTIAGA R, et al. Glasses detection on real images based on robust alignment[J]. Machine Vision and Applications, 2015, 26(4): 519–531. doi: 10.1007/s00138-015-0674-1 [13] SANDHAN T and CHOI J Y. Anti-glare: Tightly constrained optimization for eyeglass reflection removal[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 1675–1684. doi: 10.1109/CVPR.2017.182. [14] 曹志义, 牛少彰, 张继威. 基于半监督学习生成对抗网络的人脸还原算法研究[J]. 电子与信息学报, 2018, 40(2): 323–330. doi: 10.11999/JEIT170357CAO Zhiyi, NIU Shaozhang, ZHANG Jiwei. Research on face reduction algorithm based on generative adversarial nets with semi-supervised learning[J]. Journal of Electronics &Information Technology, 2018, 40(2): 323–330. doi: 10.11999/JEIT170357 [15] 毛亮, 魏颖慧, 刘三阳, 等. 一种用于细粒度人脸识别的眼镜去除方法[P]. 中国专利, 110569763A, 2019.MAO Liang, WEI Yinghui, LIU Sanyang, et al. Glasses removing method for fine-grained face recognition[P]. China Patent, 110569763A, 2019. [16] DONG Chao, CHEN C L, HE Kaiming, et al. Learning a deep convolutional network for image super-resolution[C]. The 13th European Conference on Computer Vision, Zurich, Switzerland, 2014: 184–199. doi: 10.1007/978-3-319-10593-2_13. [17] WEN Yandong, ZHANG Kaipeng, LI Zhifeng, et al. A discriminative feature learning approach for deep face recognition[C]. The 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016, 9911: 499–515. doi: 10.1007/978-3-319-46478-7_31. [18] WU Xiang, HE Ran, and SUN Zhenan. A lightened CNN for deep face representation[EB/OL]. https://arxiv.org/abs/1511.02683v1, 2015. [19] 谭小慧, 李昭伟, 樊亚春. 基于多尺度细节增强的面部表情识别方法[J]. 电子与信息学报, 2019, 41(11): 2752–2759. doi: 10.11999/EIT181088TAN Xiaohui, LI Zhaowei, and FAN Yachun. Facial expression recognition method based on multi-scale detail enhancement[J]. Journal of Electronics &Information Technology, 2019, 41(11): 2752–2759. doi: 10.11999/EIT181088 [20] DENG Weihong, HU Jiani, ZHANG Nanhai, et al. Fine-grained face verification: FGLFW database, baselines, and human-DCMN partnership[J]. Pattern Recognition, 2017, 66: 63–73. doi: 10.1016/j.patcog.2016.11.023 [21] SIMONYAN K and ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. https://arxiv.org/abs/1409.1556v6, 2015. [22] SZEGEDY C, LIU Wei, JIA Yangqing, et al. Going deeper with convolutions[C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015: 1–9. doi: 10.1109/CVPR.2015.7298594. [23] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90. [24] JIA Yangqing, SHELHAMER E, DONAHUE J, et al. Caffe: Convolutional architecture for fast feature embedding[EB/OL]. https://arxiv.org/abs/1408.5093, 2014. [25] WANG Feng, CHENG Jian, LIU Weiyang, et al. Additive margin softmax for face verification[J]. IEEE Signal Processing Letters, 2018, 25(7): 926–930. doi: 10.1109/LSP.2018.2822810 [26] DENG Jiankang, GUO Jia, XUE Niannan, et al. ArcFace: Additive angular margin loss for deep face recognition[EB/OL]. https://arxiv.org/abs/1801.07698, 2018. [27] LIU Weiyang, WEN Yandong, YU Zhiding, et al. SphereFace: Deep hypersphere embedding for face recognition[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 6738–6746. doi: 10.1109/CVPR.2017.713.