高级搜索

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于双视角时序特征融合的毫米波雷达手势数字识别研究

冯翔 刘涛 崔文卿 吴沐府 李风从 赵宜楠

冯翔, 刘涛, 崔文卿, 吴沐府, 李风从, 赵宜楠. 基于双视角时序特征融合的毫米波雷达手势数字识别研究[J]. 电子与信息学报, 2023, 45(6): 2134-2143. doi: 10.11999/JEIT220687
引用本文: 冯翔, 刘涛, 崔文卿, 吴沐府, 李风从, 赵宜楠. 基于双视角时序特征融合的毫米波雷达手势数字识别研究[J]. 电子与信息学报, 2023, 45(6): 2134-2143. doi: 10.11999/JEIT220687
FENG Xiang, LIU Tao, CUI Wenqing, WU Mufu, LI Fengcong, ZHAO Yinan. Handwriting Number Recognition Based on Millimeter-wave Radar with Dual-view Feature Fusion Network[J]. Journal of Electronics & Information Technology, 2023, 45(6): 2134-2143. doi: 10.11999/JEIT220687
Citation: FENG Xiang, LIU Tao, CUI Wenqing, WU Mufu, LI Fengcong, ZHAO Yinan. Handwriting Number Recognition Based on Millimeter-wave Radar with Dual-view Feature Fusion Network[J]. Journal of Electronics & Information Technology, 2023, 45(6): 2134-2143. doi: 10.11999/JEIT220687

基于双视角时序特征融合的毫米波雷达手势数字识别研究

doi: 10.11999/JEIT220687
基金项目: 山东省自然科学基金(ZR2019BF037)
详细信息
    作者简介:

    冯翔:男,博士,助理研究员,研究方向为毫米波雷达信号处理

    刘涛:男,硕士生,研究方向为毫米波雷达信号处理

    李风从:男,博士,助理研究员,研究方向为雷达信号处理

    赵宜楠:男,博士,教授,博士生导师,研究方向为新体制雷达技术

    通讯作者:

    冯翔 fengxiang230316@163.com

  • 中图分类号: TN951

Handwriting Number Recognition Based on Millimeter-wave Radar with Dual-view Feature Fusion Network

Funds: The Natural Science Foundation of Shandong Province (ZR2019BF037)
  • 摘要: 疫情常态化背景下,非接触式人机交互在医疗、健康领域蕴藏着巨大的应用前景,其中利用手势识别方法实现非接触式的仪器操控逐渐成为研究热点。对此,该文提出一种利用毫米波雷达双视角时序特征融合来实现手势数字识别的方法,以提高手势识别的鲁棒性和准确性。首先,该文同步采集正面、侧面视角的毫米波雷达手势数字0~9的时序回波数据;接着,对各视角的数据进行预处理,实现杂波抑制、数据压缩;随后提取两方向的距离、速度的时序特征,并就特征的时间相关性构建嵌入注意力机制的双视角时序特征融合网络(ADVFNet);最后,基于实测数据集完成了网络训练、时序特征融合、手势数字识别等步骤。实验结果表明,该文所提方法在实测数据集上识别准确率达到95%,网络收敛速度快、模型泛化能力好,与现有方法相比具有一定优势,为后续毫米波雷达人机交互提供了新思路。
  • 图  1  手势数字的雷达时序特征图

    图  2  双向LSTM网络工作原理与细胞内部结构

    图  3  ADVFNet网络结构

    图  4  雷达数据采集场景示意图

    图  5  双视角RTM融合重建0~9手势数字轨迹结果

    图  6  t-SNE可视化特征分布图

    图  7  时间注意力分布图

    图  8  双视角时序特征融合识别结果混淆矩阵

    表  1  网络具体参数说明

    类型输入尺寸输出尺寸说明
    正向输入(Postive Input)(128×128×1)
    侧向输入(Side Input)(128×128×1)
    双向LSTM(Bi-LSTM)(128×128)(128×128)units = 64
    注意力模块(Attention Module)(128×128)(128×128)
    相乘层(Multiply)(128×128);(128×128)(128×128)
    平铺层(Flatten)(128×128)16384Activation = "ReLU"
    特征融合层(Add)16384;1638416384Dropout = 0.3
    全连接层1(Dense)16384256Activation = "ReLU"
    全连接层2(Dense)25610Activation = "Softmax"
    下载: 导出CSV

    表  2  雷达参数配置

    参数数值参数数值
    发射天线数(个)1帧周期(ms)20
    接收天线数(个)4Chirp数(个)128
    带宽(GHz)2采样点数(个)64
    调频斜率(MHz/μs)50帧数(个)100
    采样率(MHz)2距离分辨率(cm)7.5
    下载: 导出CSV

    表  3  不同情况下的纵向距离与径向距离的绝对差值与相对差值

    x(cm)y=40 cmy=50 cmy=60 cm
    r(cm)AD(cm)RD(%)r(cm)AD(cm)RD(%)r(cm)AD(cm)RD(%)
    040.0000050.0000060.00000
    140.0120.0120.03150.0100.0100.02060.0080.0080.014
    240.0490.0490.12450.0400.0400.08060.0330.0330.056
    340.1120.1120.28050.0900.0900.18060.0750.0750.125
    440.1990.1990.49850.1600.1600.31960.1330.1330.222
    540.3110.3110.77850.2490.2490.49960.2080.2080.347
    640.4470.4471.11850.3590.3590.71760.2990.2990.499
    740.6070.6071.51950.4880.4880.97560.4070.4070.678
    840.7920.7921.98050.6360.6361.27260.5310.5310.885
    941.0001.0002.50050.8040.8041.60760.6710.6711.119
    1041.2311.2313.07750.9900.9901.98060.8280.8281.379
    下载: 导出CSV

    表  4  手势识别准确率(%)

    输入特征网络类型平均准确率输入特征网络类型平均准确率
    PRTMBi-LSTM84.95PVTMBi-LSTM94.55
    SRTMBi-LSTM80.34SVTMBi-LSTM74.45
    PRTM+SRTMDVFNet92.54PVTM+SVTMDVFNet90.95
    PRTMBi-LSTM+Attention88.45PVTMBi-LSTM+Attention94.60
    SRTMBi-LSTM+Attention86.59SVTMBi-LSTM+Attention87.00
    PRTM+SRTMADVFNet95.30PVTM+SVTMADVFNet96.80
    下载: 导出CSV

    表  5  在单视角下ADVFNet的识别准确率

    输入网络类型平均准确率(%)训练时长(ms/step)
    SRTM+SVTM双流卷积神经网络89.95323
    PRTM+PVTM双流卷积神经网络92.65320
    SRTM+SVTMADVFNet93.7057
    PRTM+PVTMADVFNet95.1258
    下载: 导出CSV
  • [1] LI Xinyu, HE Yuan, FIORANELLI F, et al. Semisupervised human activity recognition with radar micro-Doppler signatures[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5103112. doi: 10.1109/TGRS.2021.3090106
    [2] EROL B and AMIN M G. Radar data cube processing for human activity recognition using multisubspace learning[J]. IEEE Transactions on Aerospace and Electronic Systems, 2019, 55(6): 3617–3628. doi: 10.1109/TAES.2019.2910980
    [3] 冯心欣, 李文龙, 何兆, 等. 基于调频连续波雷达的多维信息特征融合人体姿势识别方法[J]. 电子与信息学报, 2022, 40(10): 3583–3591. doi: 10.11999JEIT220687

    FENG Xinxin, LI Wenlong, HE Zhao, et al. Multi-dimensional information feature fusion for posture recognition based on frequency modulated continuous wave radar[J]. Journal of Electronics &Information Technology, 2022, 40(10): 3583–3591. doi: 10.11999JEIT220687
    [4] ZHANG Zhenyuan, TIAN Zengshan, and ZHOU Mu. Latern: Dynamic continuous hand gesture recognition using FMCW radar sensor[J]. IEEE Sensors Journal, 2018, 18(8): 3278–3289. doi: 10.1109/JSEN.2018.2808688
    [5] SKARIA S, AL-HOURANI A, LECH M, et al. Hand-gesture recognition using two-antenna Doppler radar with deep convolutional neural networks[J]. IEEE Sensors Journal, 2019, 19(8): 3041–3048. doi: 10.1109/JSEN.2019.2892073
    [6] HAZRA S and SANTRA A. Robust gesture recognition using millimetric-wave radar system[J]. IEEE Sensors Letters, 2018, 2(4): 1–4. doi: 10.1109/lsens.2018.2882642
    [7] FAN Teng, MA Chao, GU Zhitao, et al. Wireless hand gesture recognition based on continuous-wave Doppler radar sensors[J]. IEEE Transactions on Microwave Theory and Techniques, 2016, 64(11): 4012–4020. doi: 10.1109/TMTT.2016.2610427
    [8] CRISÓSTOMO DE CASTRO FILHO H, ABÍLIO DE CARVALHO JÚNIOR O, FERREIRA DE CARVALHO O L, et al. Rice crop detection using LSTM, Bi-LSTM, and machine learning models from Sentinel-1 time series[J]. Remote Sensing, 2020, 12(16): 2655. doi: 10.3390/rs12162655
    [9] SHRESTHA A, LI Haobo, LE KERNEC J, et al. Continuous human activity classification from FMCW radar with Bi-LSTM networks[J]. IEEE Sensors Journal, 2020, 20(22): 13607–13619. doi: 10.1109/JSEN.2020.3006386
    [10] SUN Yuliang, FEI Tai, LI Xibo, et al. Real-time radar-based gesture detection and recognition built in an edge-computing platform[J]. IEEE Sensors Journal, 2020, 20(18): 10706–10716. doi: 10.1109/JSEN.2020.2994292
    [11] 王俊, 郑彤, 雷鹏, 等. 基于卷积神经网络的手势动作雷达识别方法[J]. 北京航空航天大学学报, 2018, 44(6): 1117–1123. doi: 10.13700/j.bh.1001-5965.2017.0397

    WANG Jun, ZHENG Tong, LEI Peng, et al. Hand gesture recognition method by radar based on convolutional neural network[J]. Journal of Beijing University of Aeronautics and Astronautics, 2018, 44(6): 1117–1123. doi: 10.13700/j.bh.1001-5965.2017.0397
    [12] 王勇, 王沙沙, 田增山, 等. 基于FMCW雷达的双流融合神经网络手势识别方法[J]. 电子学报, 2019, 47(7): 1408–1415. doi: 10.3969/j.issn.0372-2112.2019.07.003

    WANG Yong, WANG Shasha, TIAN Zengshan, et al. Two-stream fusion neural network approach for hand gesture recognition based on FMCW radar[J]. Acta Electronica Sinica, 2019, 47(7): 1408–1415. doi: 10.3969/j.issn.0372-2112.2019.07.003
    [13] 王勇, 吴金君, 田增山, 等. 基于FMCW雷达的多维参数手势识别算法[J]. 电子与信息学报, 2019, 41(4): 822–829. doi: 10.11999/JEIT180485

    WANG Yong, WU Jinjun, TIAN Zengshan, et al. Gesture recognition with multi-dimensional parameter using FMCW radar[J]. Journal of Electronics &Information Technology, 2019, 41(4): 822–829. doi: 10.11999/JEIT180485
    [14] 夏朝阳, 周成龙, 介钧誉, 等. 基于多通道调频连续波毫米波雷达的微动手势识别[J]. 电子与信息学报, 2020, 42(1): 164–172. doi: 10.11999/JEIT190797

    XIA Zhaoyang, ZHOU Chenglong, JIE Junyu, et al. Micro-motion gesture recognition based on multi-channel frequency modulated continuous wave millimeter wave radar[J]. Journal of Electronics &Information Technology, 2020, 42(1): 164–172. doi: 10.11999/JEIT190797
    [15] 靳标, 彭宇, 邝晓飞, 等. 基于串联式一维神经网络的毫米波雷达动态手势识别方法[J]. 电子与信息学报, 2021, 43(9): 2743–2750. doi: 10.11999/JEIT200894

    JIN Biao, PENG Yu, KUANG Xiaofei, et al. Dynamic gesture recognition method based on millimeter-wave radar by one-dimensional series neural network[J]. Journal of Electronics &Information Technology, 2021, 43(9): 2743–2750. doi: 10.11999/JEIT200894
    [16] SHEN Xiangyu, ZHENG Haifeng, FENG Xinxin, et al. ML-HGR-Net: A meta-learning network for FMCW radar based hand gesture recognition[J]. IEEE Sensors Journal, 2022, 22(11): 10808–10817. doi: 10.1109/JSEN.2022.3169231
    [17] DONG Xichao, ZHAO Zewei, WANG Yupei, et al. FMCW radar-based hand gesture recognition using spatiotemporal deformable and context-aware convolutional 5-D feature representation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5107011. doi: 10.1109/TGRS.2021.3122332
    [18] RYU S J, SUH J S, BAEK S H, et al. Feature-based hand gesture recognition using an FMCW radar and its temporal feature analysis[J]. IEEE Sensors Journal, 2018, 18(18): 7593–7602. doi: 10.1109/JSEN.2018.2859815
    [19] 邵正途, 许登荣, 徐文利, 等. 基于LSTM和残差网络的雷达有源干扰识别[J]. 系统工程与电子技术, 2023, 45(2): 416–423.

    SHAO Zhengtu, XU Dengrong, XU Wenli, et al. Radar active jamming recognition based on LSTM and residual network[J]. Systems Engineering and Electronics, 2023, 45(2): 416–423.
    [20] SHERSTINSKY A. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network[J]. Physica D:Nonlinear Phenomena, 2020, 404: 132306. doi: 10.1016/j.physd.2019.132306
    [21] JI Yuzhu, ZHANG Haijun, and WU Q M J. Salient object detection via multi-scale attention CNN[J]. Neurocomputing, 2018, 322: 130–140. doi: 10.1016/j.neucom.2018.09.061
    [22] LI Liu, XU Mai, LIU Hanruo, et al. A large-scale database and a CNN model for attention-based glaucoma detection[J]. IEEE Transactions on Medical Imaging, 2020, 39(2): 413–424. doi: 10.1109/TMI.2019.2927226
    [23] 韩崇, 韩磊, 孙力娟, 等. 基于时空压缩特征表示学习的毫米波雷达手势识别算法[J]. 电子与信息学报, 2022, 44(4): 1274–1283. doi: 10.11999/JEIT211221

    HAN Chong, HAN Lei. SUN Lijuan, et al. Millimeter wave radar gesture recognition algorithm based on Spatio-temporal compression feature representation learning[J]. Journal of Electronics &Information Technology, 2022, 44(4): 1274–1283. doi: 10.11999/JEIT211221
  • 加载中
图(8) / 表(5)
计量
  • 文章访问数:  576
  • HTML全文浏览量:  288
  • PDF下载量:  122
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-05-27
  • 修回日期:  2022-10-05
  • 网络出版日期:  2022-10-11
  • 刊出日期:  2023-06-10

目录

    /

    返回文章
    返回