基于动态视觉运动特征的脉冲神经网络识别方法

董峻妃; 姜润皓; 燕锐; 唐华锦

doi:10.11999/JEIT221478

基于动态视觉运动特征的脉冲神经网络识别方法

doi: 10.11999/JEIT221478

1.
四川大学计算机学院成都 610065
2.
浙江大学计算机科学与技术学院杭州 310058
3.
浙江工业大学计算机科学与技术学院杭州 310014

基金项目: 国家重点研发计划 (2020AAA0105900)

详细信息

作者简介:
董峻妃：女，博士生，研究方向为类脑计算、脉冲神经网络以及神经形态视觉

姜润皓：男，博士生，研究方向为类脑计算、脉冲神经网络

燕锐：女，教授，研究方向为类脑计算、神经计算与智能机器人

唐华锦：男，教授，研究方向为类脑计算、计算神经科学以及智能机器人

通讯作者:
唐华锦　htang@zju.edu.cn

中图分类号: TP391.4
计量
- 文章访问数: 745
- HTML全文浏览量: 320
- PDF下载量: 191
- 被引次数: 0
出版历程
- 收稿日期: 2022-11-25
- 修回日期: 2023-05-02
- 网络出版日期: 2023-05-19
- 刊出日期: 2023-08-21

Spiking Neural Network Recognition Method Based on Dynamic Visual Motion Features

1.
College of Computer Science, Sichuan University, Chengdu 610065, China
2.
College of Computer Science and Technology, Zhejiang University, Hangzhou 310058, China
3.
College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310014, China

Funds: The National Key Research and Development Program of China (2020AAA0105900)

摘要

摘要: 针对现有脉冲神经网络(SNN)对动态视觉事件流识别精度低与实时性差等问题，该文提出一种基于动态视觉运动特征的脉冲神经网络识别方法。首先利用基于事件的运动历史信息表示与梯度方向计算提取事件流中的动态运动特征；然后引入时空池化操作来消除事件在时间和空间上的冗余，保留显著的运动特征；最后，将特征事件流输入脉冲神经网络进行学习与识别。在基准的动态视觉数据集上的实验结果表明，动态视觉运动特征可显著提升SNN对于事件流的识别精度与计算速度。
- 动态视觉感知 /
- 事件相机 /
- 脉冲神经网络 /
- 动作识别 /
- 运动特征提取
Abstract: Considering the shortcomings of the low recognition accuracy and poor real-time performance of existing Spiking Neural Networks (SNN) for dynamic visual event streams, a SNN recognition method based on dynamic visual motion features is proposed in this paper. First, the dynamic motion features in the event stream are extracted using the event-based motion history information representation and gradient direction calculation. Then, the spatiotemporal pooling operation is introduced to eliminate the redundancy of events in the temporal and spatial domain, further retaining the significant motion features. Finally, the feature event streams are fed into the SNN for learning and recognition. Experiments conducted on benchmark dynamic visual datasets show that dynamic visual motion features can significantly improve the recognition accuracy and computational speed of SNN for event streams.
- Dynamic visual perception /
- Event camera /
- Spiking Neural Network (SNN) /
- Action recognition /
- Motion feature extraction

HTML全文

图 1 基于动态视觉运动特征的SNN识别方法流程图

下载: 全尺寸图片幻灯片

图 2 动态视觉特征表示方法对比图

下载: 全尺寸图片幻灯片

图 3 不同特征表示方法下的识别混淆矩阵结果比较

下载: 全尺寸图片幻灯片

算法1 动态视觉运动特征算法
1: 输出: $ ({e_i},{\theta _{{e_i}}}) $
2: 输入: $ r,\tau $，初始化$ {\bf{IS}}(x,y,p) \leftarrow 0 $
3: for 每个到来的事件 ${{\boldsymbol{e}}_i}$ do
4: 　$ {\bf{IS}}({x_i},{y_i},{p_i}) \leftarrow i $ // 更新$ {\bf{IS}} $
5: 　计算十字邻域$U({{\boldsymbol{e}}_i},r)$，式(3)—式(6)
6: 　利用式(8)、式(9)计算$ \Delta X $和$ \Delta Y $
7: 　计算当前事件的运动梯度方向$ {\theta _{{e_i}}} $，式(7)—式(10)
8: 　$ ({x_i},{y_i}) \leftarrow ({x_i}/4,{y_i}/4) $ // 时空池化
9: 　if ${{\boldsymbol{e}}_i}$不在不应期内 then
10: 　输出$({{\boldsymbol{e}}_i},{\theta _{ {e_i} } })$
11: 　end if
12: end for

下载: 导出CSV

表 1 数据集信息与划分情况

数据集	类别数	分辨率	训练集	测试集
DVS128 Gesture	11	128×128	1151	191
Action Recognition	10	346×260	250	41
DailyAction-DVS	12	128×128	1235	205

下载: 导出CSV

表 2 基于事件的动作识别算法比较(%)

方法	结构	DVS128 Gesture	Action Recognition	DailyAction-DVS
SLAYER^[12]	深度SNN (8层)	93.6	–	–
STBP^[13]	深度SNN (6层)	93.4	–	–
DECOLLE^[14]	深度SNN (6层)	95.5	79.6	91.7
SCRNN^[21]	脉冲卷积循环网络(5层)	92.0	–	–
文献[22]	基于卷积和储层计算的SNN	65.0	–	–
文献[9]	HFirst特征+单层SNN	61.5	55.0	68.3
SPA ^[18]	HFirst特征+单层SNN	70.1	–	76.9
文献[15]	光流特征+单层SNN	92.7	78.1	90.3
本文	运动特征+单层SNN	94.7	79.5	96.1
注：加粗字体表示各列最优结果；“–”表示此处数据为空

下载: 导出CSV

表 3 不同特征表示方法的性能比较

特征表示方法	识别精度(%)	处理速度(kev/s)
HFirst特征	61.50	32.9
HOTS特征	85.56	54.7
光流运动特征	91.53	3.67
运动特征(本文)	94.65	913
注：加粗字体表示各列最优结果

下载: 导出CSV

表 4 不同运动历史信息表示方法的性能比较(%)

运动历史信息	时间平面	计数平面	索引平面
DVS128 Gesture	93.58	91.44	94.65
Action Recognition	76.92	74.36	79.49
DailyAction -DVS	94.63	93.66	96.10
注：加粗字体表示各列最优结果

下载: 导出CSV

参考文献(24)

[1]	GALLEGO G, DELBRÜCK T, ORCHARD G, et al. Event-based vision: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 154–180. doi: 10.1109/TPAMI.2020.3008413
[2]	张宇豪, 袁孟雯, 陆宇婧, 等. 面向动态事件流的神经网络转换方法[J]. 计算机应用, 2022, 42(10): 3033–3039. doi: 10.11772/j.issn.1001-9081.2021091607 ZHANG Yuhao, YUAN Mengwen, LU Yujing, et al. Neural network conversion method for dynamic event stream[J]. Journal of Computer Applications, 2022, 42(10): 3033–3039. doi: 10.11772/j.issn.1001-9081.2021091607
[3]	李家宁, 田永鸿. 神经形态视觉传感器的研究进展及应用综述[J]. 计算机学报, 2021, 44(6): 1258–1286. LI Jianing and TIAN Yonghong, Recent advances in neuromorphic vision sensors: A survey[J]. Chinese Journal of Computers, 2021, 44(6): 1258–1286.
[4]	ROY K, JAISWAL A, and PANDA P. Towards spike-based machine intelligence with neuromorphic computing[J]. Nature, 2019, 575(7784): 607–617. doi: 10.1038/s41586-019-1677-2
[5]	张铁林, 徐波. 脉冲神经网络研究现状及展望[J]. 计算机学报, 2021, 44(9): 1767–1785. ZHANG Tielin and XU Bo. Research advances and perspectives on spiking neural networks[J]. Chinese Journal of Computers, 2021, 44(9): 1767–1785.
[6]	胡一凡, 李国齐, 吴郁杰, 等. 脉冲神经网络研究进展综述[J]. 控制与决策, 2021, 36(1): 1–26. doi: 10.13195/j.kzyjc.2020.1006 HU Yifan, LI Guoqi, WU Yujie, et al. Spiking neural networks: A survey on recent advances and new directions[J]. Control and Decision, 2021, 36(1): 1–26. doi: 10.13195/j.kzyjc.2020.1006
[7]	ORCHARD G, MEYER C, ETIENNE-CUMMINGS R, et al. HFirst: A temporal approach to object recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(10): 2028–2040. doi: 10.1109/TPAMI.2015.2392947
[8]	ZHAO Bo, DING Ruoxi, CHEN Shoushun, et al. Feedforward categorization on AER motion events using cortex-like features in a spiking neural network[J]. IEEE Transactions on Neural Networks and Learning Systems, 2015, 26(9): 1963–1978. doi: 10.1109/TNNLS.2014.2362542
[9]	XIAO Rong, TANG Huajin, MA Yuhao, et al. An event-driven categorization model for AER image sensors using multispike encoding and learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(9): 3649–3657. doi: 10.1109/TNNLS.2019.2945630
[10]	LIU Qianhui, PAN Gang, RUAN Haibo, et al. Unsupervised AER object recognition based on multiscale spatio-temporal features and spiking neurons[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(12): 5300–5311. doi: 10.1109/TNNLS.2020.2966058
[11]	LIU Qianhui, RUAN Haibo, XING Dong, et al. Effective AER object classification using segmented probability-maximization learning in spiking neural networks[C]. The 34th AAAI Conference on Artificial Intelligence, New York, USA, 2020: 1308–1315.
[12]	HE Weihua, WU Yujie, DENG Lei, et al. Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences[J]. Neural Networks, 2020, 132: 108–120. doi: 10.1016/j.neunet.2020.08.001
[13]	SHRESTHA S B and ORCHARD G. SLAYER: Spike layer error reassignment in time[C]. The 32nd International Conference on Neural Information Processing Systems, Montreal, Canada, 2018: 1419–1428.
[14]	KAISER J, MOSTAFA H, and NEFTCI E. Synaptic plasticity dynamics for deep continuous local learning (DECOLLE)[J]. Frontiers in Neuroscience, 2020, 14: 424. doi: 10.3389/fnins.2020.00424
[15]	LIU Qianhui, XING Dong, TANG Huajin, et al. Event-based action recognition using motion information and spiking neural networks[C]. The Thirtieth International Joint Conference on Artificial Intelligence, Montreal, Canada, 2021: 1743–1749.
[16]	AFSHAR S, HAMILTON T J, TAPSON J, et al. Investigation of event-based surfaces for high-speed detection, unsupervised feature extraction, and object recognition[J]. Frontiers in Neuroscience, 2019, 12: 1047. doi: 10.3389/fnins.2018.01047
[17]	MANDERSCHEID J, SIRONI A, BOURDIS N, et al. Speed invariant time surface for learning to detect corner points with event-based cameras[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 10237–10246.
[18]	ZHAO Bo, YU Qiang, DING Ruoxi, et al. Event-driven simulation of the tempotron spiking neuron[C]. 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS) Proceedings, Lausanne, Switzerland, 2014: 667–670.
[19]	AMIR A, TABA B, BERG D, et al. A low power, fully event-based gesture recognition system[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 7388–7397.
[20]	MIAO Shu, CHEN Guang, NING Xiangyu, et al. Neuromorphic vision datasets for pedestrian detection, action recognition, and fall detection[J]. Frontiers in Neurorobotics, 2019, 13: 38. doi: 10.3389/fnbot.2019.00038
[21]	XING Yannan, DI CATERINA G, and SORAGHAN J. A new spiking convolutional recurrent neural network (SCRNN) with applications to event-based hand gesture recognition[J]. Frontiers in Neuroscience, 2020, 14: 590164. doi: 10.3389/fnins.2020.590164
[22]	GEORGE A M, BANERJEE D, DEY S, et al. A reservoir-based convolutional spiking neural network for gesture recognition from DVS input[C]. 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, United Kingdom, 2020: 1–9.
[23]	LU Junwei, DONG Junfei, YAN Rui, et al. An event-based categorization model using spatio-temporal features in a spiking neural network[C]. 2020 12th International Conference on Advanced Computational Intelligence (ICACI), Dali, China, 2020: 385–390.
[24]	LAGORCE X, ORCHARD G, GALLUPPI F, et al. HOTS: A hierarchy of event-based time-surfaces for pattern recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(7): 1346–1359. doi: 10.1109/TPAMI.2016.2574707