一种用于细粒度人脸识别的眼镜去除方法

毛亮; 薛月菊; 魏颖慧; 朱婷婷

doi:10.11999/JEIT200176

一种用于细粒度人脸识别的眼镜去除方法

doi: 10.11999/JEIT200176

1.
深圳职业技术学院粤港澳大湾区人工智能应用技术研究院深圳 518055
2.
华南农业大学电子工程学院广州 510642
3.
西安电子科技大学数学与统计学院西安 710071

基金项目: 国家科技支撑计划(2015BAD06B03-3)

详细信息

作者简介:
毛亮：男，1983年生，副研究员，研究方向计算机视觉与深度学习

薛月菊：女，1969年生，教授，主要研究方向机器视觉与图像处理计算机视觉与深度学习

通讯作者:
薛月菊　xueyueju@163.com

中图分类号: TN911.73; TP391.4
计量
- 文章访问数: 2701
- HTML全文浏览量: 1498
- PDF下载量: 116
- 被引次数: 0
出版历程
- 收稿日期: 2020-03-17
- 修回日期: 2020-07-20
- 网络出版日期: 2020-07-27
- 刊出日期: 2021-05-18

An Eyeglasses Removal Method for Fine-grained Face Recognition

1.
Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong-Macao Greater Bay Area, Shenzhen Polytechnic, Shenzhen 518055, China
2.
College of Electronic Engineering, South China Agricultural University, Guangzhou 510642, China
3.
School of Mathematics and Statistics, Xidian University, Xi’an 710071, China

Funds: The National Science and Technology Support Program (2015BAD06B03-3)

摘要

摘要: 为解决眼镜遮挡会降低人脸识别性能的难点，借鉴深度卷积神经网络在超分辨率方面的成功应用，该文提出一种用于细粒度人脸识别的眼镜自动去除方法ERCNN。用卷积层、池化层、MFM特征选取模块和反卷积层设计ERCNN网络模型，自动学习戴眼镜和未戴眼镜人脸图像对之间的映射关系，实现端到端的眼镜去除。然后，收集大量监控场景下的人脸图像，以及互联网上公开的人脸图像作为训练集；同时构建SLLFW数据集，作为眼镜去除和人脸识别的测试集。最后，通过与传统的眼镜去除方法进行对比试验，该文算法的各项评价指标优于传统方法，能有效的去除真实人脸图像中眼镜；同时在SLLFW人脸数据集上形成的全框眼镜、半框眼镜和无框眼镜人脸数据集上对多种人脸识别算法进行对比试验。试验表明，在FAR为1%的情况下，利用该文方法对F-SLLFW, H-SLLFW和R-SLLFW数据集的人脸图像进行眼镜去除后，SphereFace算法的TAR分别达到90.05%, 91.14%和92.33%，比未去除眼镜的识别率分别提高了3.92%, 3.08%和1.26%；同样，在FAR为0.1%的情况下，比SphereFace算法的TAR分别提高了10.06%, 4.29%和2.13%，说明该文方法有助于提升细粒度人脸识别的识别精度。
- 人脸识别 /
- 深度卷积网络 /
- 特征选择 /
- 眼镜去除
Abstract: In order to solve the problem that eyeglasses reduce often the performance of face recognition, based on the successful application of deep convolution neural network in super-resolution, This paper proposes an automatic eyeglasses removal method ERCNN (Eyeglasses Removal CNN) for fine-grained face recognition. Specifically, the ERCNN network which is designed based on the convolution layer, pool layer, MFM (Max Feature Map)feature selection module and deconvolution layer, are automatically learned the mapping relationship between facial images with eyeglasses and their counterparts without eyeglasses to realize end-to-end eyeglasses removal. Then, massive facial images are captured through surveillance equipment and collected from the Internet as the training set. And, SLLFW data set is established, which is used as the test set of eyeglasses removal and face recognition. The experiment show that the proposed method can better effectively remove the eyeglasses from the real facial image than the traditional eyeglasses removal methods, and the evaluation index of the method is better than other methods. In addition, several face recognition methods are tested separately on the facial images formed by SLLFW data set. Experiments show that when the FAR (False Accept Rate) is 1%, the TAR (True Accept Rate) of the Sphereface method reaches 90.05%, 91.14% and 92.33%, which is 3.92%, 3.08% and 1.26% higher than the Sphereface method is not used to remove the eyeglasses from the F-SLLFW, H-SLLF and R-SLLFW, respectively. Similarly, when the FAR 0.1%, the TAR of Sphereface method is increased by 10.06%, 3.08% and 1.26% respectively. Therefore, the proposed method can better improve the recognition accuracy of fine-grained face recognition.
- Face recognition /
- Deep Convolution Neural Network (DCNN) /
- Feature selection /
- Eyeglasses removal

HTML全文

图 1 戴不同类型眼镜的人脸图像

下载: 全尺寸图片幻灯片

图 2 人脸数据集

下载: 全尺寸图片幻灯片

图 3 ERCNN的网络结构

Conv2a, ···, Conv2d, Conv4为卷积层；ReLU3为激活函数

下载: 全尺寸图片幻灯片

图 4 人脸眼镜去除的结果

下载: 全尺寸图片幻灯片

图 5 真实戴眼镜人脸的眼镜去除的结果

下载: 全尺寸图片幻灯片

图 6 真实戴眼镜人脸的眼镜去除效果不佳的结果

下载: 全尺寸图片幻灯片

表 1 不同方法以及不同类型的眼镜去除性能对比

人脸对齐	眼镜类型	方法	评价指标
人脸对齐	眼镜类型	方法	PSNR (dB)	SSIM	MSE	IFC
128×128	全框	ERCNN	34.7	0.98	22.03	7.0
	全框	PCA	24.2	0.85	247.22	6.1
	半框	ERCNN	35.7	0.99	17.5	7.3
	半框	PCA	25.5	0.87	183.27	6.2
	无框	ERCNN	35.7	0.99	17.5	7.5
	无框	PCA	26.0	0.88	163.34	6.3
	平均值	ERCNN	35.3	0.98	19.01	7.2
	平均值	PCA	25.2	0.86	197.94	6.2
112×96	全框	ERCNN	35.6	0.99	17.91	7.3
	全框	PCA	24.3	0.84	241.59	4.5
	半框	ERCNN	36.3	0.99	15.24	7.6
	半框	PCA	25.0	0.85	205.63	4.5
	无框	ERCNN	36.1	0.99	15.96	7.7
	无框	PCA	24.9	0.85	210.42	4.6
	平均值	ERCNN	36.0	0.99	16.37	7.5
	平均值	PCA	24.7	0.84	219.21	4.5

下载: 导出CSV

表 2 不同人脸识别的识别性能对比(%)

数据集	FAR	TAR
数据集	FAR	①	②	③	④
LFW	1	99.03	98.13	99.07	99.40
LFW	0.1	95.50	91.27	95.90	97.04
SLLFW	1	93.17	78.43	92.53	96.11
SLLFW	0.1	84.33	62.23	85.43	91.38
F-SLLFW	1	80.87	55.77	83.20	86.13
F-SLLFW	0.1	68.27	39.87	70.70	71.73
本文方法对F-SLLFW进行处理后	1	86.47	64.47	88.63	90.05
本文方法对F-SLLFW进行处理后	0.1	77.07	48.10	77.87	81.79
H-SLLFW	1	83.10	60.70	86.27	88.06
H-SLLFW	0.1	72.80	43.53	75.10	78.36
本文方法对H-SLLFW进行处理后	1	87.60	67.07	89.93	91.14
本文方法对H-SLLFW进行处理后	0.1	80.70	45.53	78.83	82.65
R-SLLFW	1	87.07	66.10	89.37	91.70
R-SLLFW	0.1	79.53	49.67	76.73	82.76
本文方法对R-SLLFW进行处理后	1	88.53	67.50	89.13	92.33
本文方法对R-SLLFW进行处理后	0.1	81.07	47.80	77.43	84.89

下载: 导出CSV

参考文献(27)

[1]	李小薪, 梁荣华. 有遮挡人脸识别综述: 从子空间回归到深度学习[J]. 计算机学报, 2018, 41(1): 177–207. doi: 10.11897/SP.J.1016.2018.00177 LI Xiaoxin and LIANG Ronghua. A review for face recognition with occlusion: From subspace regression to deep learning[J]. Chinese Journal of Computers, 2018, 41(1): 177–207. doi: 10.11897/SP.J.1016.2018.00177
[2]	WU Chenyu, LIU Ce, SHUM H Y, et al. Automatic eyeglasses removal from face images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(3): 322–336. doi: 10.1109/TPAMI.2004.1262319
[3]	DU Cheng and SU Guangda. Eyeglasses removal from facial images[J]. Pattern Recognition Letters, 2005, 26(14): 2215–2220. doi: 10.1016/j.patrec.2005.04.002
[4]	PARK J S, OH Y H, AHN S C, et al. Glasses removal from facial image using recursive error compensation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(5): 805–811. doi: 10.1109/TPAMI.2005.103
[5]	HEO J and SAVVIDES M. Face pose correction with eyeglasses and occlusions removal[C]. 2007 IEEE Conference Biometrics Symposium, Baltimore, USA, 2007: 1–6. doi: 10.1109/BCC.2007.4430551.
[6]	JIA Xiaodong and GUO Jiangling. Eyeglasses removal from facial image based on phase congruency[C]. The 3rd IEEE International Congress on Image and Signal Processing, Yantai, China, 2010: 1859–1862, doi: 10.1109/CISP.2010.5647366.
[7]	WANG Yuankai, JANG J H, TSAI L W, et al. Improvement of face recognition by eyeglass removal[C]. The 6th IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Darmstadt, Germany, 2010: 228–231. doi: 10.1109/IIHMSP.2010.64.
[8]	CHENG Wenchang, LIAO H C, PAN M H, et al. A fatigue detection system with eyeglasses removal[C]. The 15th IEEE International Conference on Advanced Communications Technology, PyeongChang, South Korea, 2013: 331–335.
[9]	李根, 李文辉. 基于思维进化的机器学习的遮挡人脸识别[J]. 吉林大学学报: 工学版, 2014, 44(5): 1410–1416. LI Gen and LI Wenhui. Face occlusion recognition based on MEBML[J]. Journal of Jilin University:Engineering and Technology Edition, 2014, 44(5): 1410–1416.
[10]	GUO Pei and SU Fei. Enhanced PCA reconstruction method for eyeglass frame auto-removal[C]. The 4th IEEE International Conference on Network Infrastructure and Digital Content, Beijing, China, 2014: 359–363. doi: 10.1109/ICNIDC.2014.7000325.
[11]	吴从中, 刘渠芬, 詹曙. 基于稀疏表达的人脸遮挡物去除[J]. 合肥工业大学学报: 自然科学版, 2015, 38(5): 615–617, 680. WU Congzhong, LIU Qufen, and ZHAN Shu. A face occlusion removal method based on sparse representation[J]. Journal of Hefei University of Technology, 2015, 38(5): 615–617, 680.
[12]	FERNÁNDEZ A, GARCÍA R, USAMENTIAGA R, et al. Glasses detection on real images based on robust alignment[J]. Machine Vision and Applications, 2015, 26(4): 519–531. doi: 10.1007/s00138-015-0674-1
[13]	SANDHAN T and CHOI J Y. Anti-glare: Tightly constrained optimization for eyeglass reflection removal[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 1675–1684. doi: 10.1109/CVPR.2017.182.
[14]	曹志义, 牛少彰, 张继威. 基于半监督学习生成对抗网络的人脸还原算法研究[J]. 电子与信息学报, 2018, 40(2): 323–330. doi: 10.11999/JEIT170357 CAO Zhiyi, NIU Shaozhang, ZHANG Jiwei. Research on face reduction algorithm based on generative adversarial nets with semi-supervised learning[J]. Journal of Electronics &Information Technology, 2018, 40(2): 323–330. doi: 10.11999/JEIT170357
[15]	毛亮, 魏颖慧, 刘三阳, 等. 一种用于细粒度人脸识别的眼镜去除方法[P]. 中国专利, 110569763A, 2019. MAO Liang, WEI Yinghui, LIU Sanyang, et al. Glasses removing method for fine-grained face recognition[P]. China Patent, 110569763A, 2019.
[16]	DONG Chao, CHEN C L, HE Kaiming, et al. Learning a deep convolutional network for image super-resolution[C]. The 13th European Conference on Computer Vision, Zurich, Switzerland, 2014: 184–199. doi: 10.1007/978-3-319-10593-2_13.
[17]	WEN Yandong, ZHANG Kaipeng, LI Zhifeng, et al. A discriminative feature learning approach for deep face recognition[C]. The 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016, 9911: 499–515. doi: 10.1007/978-3-319-46478-7_31.
[18]	WU Xiang, HE Ran, and SUN Zhenan. A lightened CNN for deep face representation[EB/OL]. https://arxiv.org/abs/1511.02683v1, 2015.
[19]	谭小慧, 李昭伟, 樊亚春. 基于多尺度细节增强的面部表情识别方法[J]. 电子与信息学报, 2019, 41(11): 2752–2759. doi: 10.11999/EIT181088 TAN Xiaohui, LI Zhaowei, and FAN Yachun. Facial expression recognition method based on multi-scale detail enhancement[J]. Journal of Electronics &Information Technology, 2019, 41(11): 2752–2759. doi: 10.11999/EIT181088
[20]	DENG Weihong, HU Jiani, ZHANG Nanhai, et al. Fine-grained face verification: FGLFW database, baselines, and human-DCMN partnership[J]. Pattern Recognition, 2017, 66: 63–73. doi: 10.1016/j.patcog.2016.11.023
[21]	SIMONYAN K and ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. https://arxiv.org/abs/1409.1556v6, 2015.
[22]	SZEGEDY C, LIU Wei, JIA Yangqing, et al. Going deeper with convolutions[C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015: 1–9. doi: 10.1109/CVPR.2015.7298594.
[23]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.
[24]	JIA Yangqing, SHELHAMER E, DONAHUE J, et al. Caffe: Convolutional architecture for fast feature embedding[EB/OL]. https://arxiv.org/abs/1408.5093, 2014.
[25]	WANG Feng, CHENG Jian, LIU Weiyang, et al. Additive margin softmax for face verification[J]. IEEE Signal Processing Letters, 2018, 25(7): 926–930. doi: 10.1109/LSP.2018.2822810
[26]	DENG Jiankang, GUO Jia, XUE Niannan, et al. ArcFace: Additive angular margin loss for deep face recognition[EB/OL]. https://arxiv.org/abs/1801.07698, 2018.
[27]	LIU Weiyang, WEN Yandong, YU Zhiding, et al. SphereFace: Deep hypersphere embedding for face recognition[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 6738–6746. doi: 10.1109/CVPR.2017.713.