A Processing-In-Memory Neural Network Inference System Design for Infrared Gesture Recognition
-
摘要: 红外识别在安防监控、人机交互和无人系统中具有重要应用，但红外图像对比度低、边缘模糊等特性，导致传统算法难以在资源受限的边缘端兼顾识别精度与实时性。为突破深度学习算法在边缘端部署时面临的高算力需求与高能耗预算之间的矛盾，该文设计一种基于忆阻器阵列加速的红外识别系统，将两层全连接网络的前向推理映射到忆阻器阵列中执行，并通过计算机实现层间非线性激活与最终判决，从而有效减轻传统CPU/GPU架构下的数据搬迁开销。基于该架构，该文搭建了可运行的3分类红外手势识别系统。实验结果表明，该系统在低分辨率输入下仍能获得约95%的识别准确率，推理延迟约26.4 ms，能耗相较嵌入式GPU平台降低约6倍，体现出显著的能效优势和实时性潜力。本研究为构建高能效的红外边缘智能系统提供了一种可行思路。Abstract:
Objective Infrared (IR) image recognition is widely used in security monitoring, autonomous driving, and human-machine interaction, where stable perception under low illumination and noisy backgrounds is required. However, IR images often exhibit low contrast, blurred edges, and weak texture representation, which reduce recognition accuracy. Conventional Deep Neural Networks (DNNs) further aggravate this issue in edge environments because the von Neumann architecture requires intensive computation, frequent memory-processor data transfer, and high power consumption. Memristor devices provide an alternative because they support in-memory computing. However, most existing implementations still depend on CPUs or GPUs to execute nonlinear operations such as activation layers, which reintroduces data transfer overhead. To address this issue, a nearly all-memristor neural network framework for IR image recognition is proposed. In this framework, computationally dominant linear operations are executed entirely on memristor arrays, whereas CPU participation is limited to the final activation step. Methods The proposed system maps two fully connected layers of a neural network directly onto memristor crossbar arrays. This mapping enables large-scale matrix-vector multiplications to be executed in memory with high parallelism and low energy consumption. Network weights are encoded as memristor conductance states, and inference is performed by applying voltage inputs and summing output currents. Intermediate nonlinear activations and final classification are computed on the host computer. Because these operations require minimal computation, the CPU overhead in both latency and energy remains negligible. Based on this design, an IR gesture recognition system is constructed to distinguish two hand gestures and a no-gesture state. System evaluation considers recognition accuracy, inference latency, energy consumption, and the amount of data transferred between the memristor arrays and the CPU. A CPU-only neural network is used as the baseline. The framework provides a practical approach for near all-hardware neural network computing while maintaining recognition accuracy and reducing energy consumption and data transfer for edge-scale IR applications. Results and Discussions The system is evaluated on a three-class IR gesture recognition task that includes two gestures and a no-gesture state. The memristor-based network achieves approximately 95% accuracy ( Table 2 ), which is close to the 97% obtained with a GPU implementation (Table 2 ). This result indicates that conductance variation has limited effect on recognition performance. The average inference latency per frame is substantially lower than that of the GPU baseline, and the achieved frame rate satisfies real-time requirements. Power measurements indicate that the memristor array consumes only milliwatts, whereas the GPU requires approximately 25 W (Table 2 ). Although peripheral circuit consumption is not fully included, the results demonstrate the inherent energy efficiency of in-memory computing. Intermediate and final activation outputs are transferred to the CPU, which removes most memory-processor interactions. This reduction in data movement, combined with array-level parallelism, accounts for the observed improvements in latency and energy consumption. Overall, the framework maintains high recognition accuracy while significantly improving computational efficiency, which indicates strong potential for edge-scale IR recognition.Conclusions This study presents a nearly all-memristor infrared neural network framework in which two fully connected layers are executed on memristor arrays and intermediate nonlinear activations are processed on the host computer. When applied to a three-class IR gesture recognition task, the system achieves recognition accuracy comparable to that of a GPU platform while significantly reducing inference latency and energy consumption. By minimizing memory-processor data transfer and exploiting in-memory computing, the framework provides clear advantages for edge applications. The results confirm the feasibility of deploying memristor-based neural networks in practical infrared recognition systems. Future research will focus on integrating memristor-based activation functions, scaling the system to larger circuits, and extending the approach to more complex network architectures and datasets. -
表 2 性能指标对比表
评价指标 Jetson Nano 忆阻器平台 系统功耗(mW) 2.5×104 11.2(阵列功耗) 平均推理延迟(ms) 161.2 26.4 平均推理准确率(%) 97 95 能效比(fps/W) 0.2 3.4×103 
下载: 导出CSV
-
[1] WANG Qingwang, JIN Pengcheng, WU Yuhang, et al. Infrared image enhancement: A review[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2025, 18: 3281–3299. doi: 10.1109/JSTARS.2024.3523418. [2] HU Lei, HU Long, and CHEN Minghui. Edge-enhanced infrared image super-resolution reconstruction model under transformer[J]. Scientific Reports, 2024, 14(1): 15585. doi: 10.1038/s41598-024-66302-8. [3] LIU Zhenhui. Intelligent classification of computer vulnerabilities and network security management system: Combining memristor neural network and improved TCNN model[J]. PLoS One, 2025, 20(1): e0318075. doi: 10.1371/journal.pone.0318075. [4] AGUIRRE F, SEBASTIAN A, LE GALLO M, et al. Hardware implementation of memristor-based artificial neural networks[J]. Nature Communications, 2024, 15(1): 1974. doi: 10.1038/s41467-024-45670-9. [5] DING Chenchen, REN Yuan, LIU Zhengwu, et al. Transforming memristor noises into computational innovations[J]. Communications Materials, 2025, 6(1): 149. doi: 10.1038/s43246-025-00876-2. [6] 张睿, 李允臣, 王家宝, 等. 基于深度学习的红外目标检测综述[J]. 计算机技术与发展, 2023, 33(11): 1–8. doi: 10.3969/j.issn.1673-629X.2023.11.001.ZHANG Rui, LI Yunchen, WANG Jiabao, et al. Survey on infrared object detection based on deep learning[J]. Computer Technology and Development, 2023, 33(11): 1–8. doi: 10.3969/j.issn.1673-629X.2023.11.001. [7] 郭昕婕, 王光燿, 王绍迪. 存内计算芯片研究进展及应用[J]. 电子与信息学报, 2023, 45(5): 1888–1898. doi: 10.11999/JEIT220420.GUO Xinjie, WANG Guangyao, and WANG Shaodi. Technology developments and applications of in-memory computing processors[J]. Journal of Electronics & Information Technology, 2023, 45(5): 1888–1898. doi: 10.11999/JEIT220420. [8] KHOR W, CHEN Y K, ROBERTS M, et al. Automated detection and classification of concealed objects using infrared thermography and convolutional neural networks[J]. Scientific Reports, 2024, 14(1): 8353. doi: 10.1038/s41598-024-56636-8. [9] YE Longcheng, GAO Zhixuan, FU Jinke, et al. Overview of memristor-based neural network design and applications[J]. Frontiers in Physics, 2022, 10: 839243. doi: 10.3389/fphy.2022.839243. [10] JIN Boyan, WANG Zhenlong, WANG Tianyu, et al. Memristor-based artificial neural networks for hardware neuromorphic computing[J]. Research, 2025, 8: 0758. doi: 10.34133/research.0758. [11] FENG Weibo, QIN Tianling, and TANG Xin. Advances in infrared detectors for in-memory sensing and computing[J]. Photonics, 2024, 11(12): 1138. doi: 10.3390/photonics11121138. [12] WANG Jiejun, PAN Xinqiang, ZHAO Zebin, et al. An infrared near-sensor reservoir computing system based on large-dynamic-space memristor with tens of thousands of states for dynamic gesture perception[J]. Advanced Science, 2024, 11(6): 2307359. doi: 10.1002/advs.202307359. [13] ZHANG Wenbin, YAO Peng, GAO Bin, et al. Edge learning using a fully integrated neuro-inspired memristor chip[J]. Science, 2023, 381(6663): 1205–1211. doi: 10.1126/science.ade3483. [14] BAE B, PARK M, LEE D, et al. Hetero-integrated InGaAs photodiode and oxide memristor-based artificial optical nerve for in-sensor NIR image processing[J]. Advanced Optical Materials, 2023, 11(3): 2201905. doi: 10.1002/adom.202201905. [15] PAN Youjiang, WU Chunyan, BU Guohao, et al. All-in-one neuromorphic light-emitting memristor for infrared image visualization and recognition[J]. Advanced Functional Materials, 2025, 35(37): 2505174. doi: 10.1002/adfm.202505174. [16] SHI Tuo, GAO Lili, TIAN Yang, et al. Memristor-based feature learning for pattern classification[J]. Nature Communications, 2025, 16(1): 913. doi: 10.1038/s41467-025-56286-y. [17] AZIZA H. Image classification in memristor-based neural networks: A comparative study of software and hardware models using RRAM crossbars[J]. Electronics, 2025, 14(6): 1125. doi: 10.3390/electronics14061125. [18] SHI Tuo, ZHANG Hui, CUI Shiyu, et al. Stochastic neuro-fuzzy system implemented in memristor crossbar arrays[J]. Science Advances, 2024(10): eadl3135. doi: 10.1126/sciadv.adl3135. [19] LO Y C, WU Junshen, WANG C C, et al. ISSA: Architecting CNN accelerators using input-skippable, set-associative computing-in-memory[J]. IEEE Transactions on Computers, 2024, 73(9): 2136–2149. doi: 10.1109/TC.2024.3404060. [20] FORNT J, FONTOVA-MUSTÉ P, CARO M, et al. An energy-efficient GeMM-based convolution accelerator with on-the-fly im2col[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2023, 31(11): 1874–1878. doi: 10.1109/TVLSI.2023.3286122. [21] 刘津畅, 时拓. 一种多值阻变存储器的振荡收敛编程方法及装置[P]. 中国, 118760422B, 2024.LIU Jinchang and SHI Tuo. Oscillation convergence programming method and device of multi-valued resistive random access memory[P]. CN, 118760422B, 2024. [22] JOSHI V, LE GALLO M, HAEFELI S, et al. Accurate deep neural network inference using computational phase-change memory[J]. Nature Communications, 2020, 11(1): 2473. doi: 10.1038/s41467-020-16108-9. -
下载:
图(5) / 表(2)
计量
- 文章访问数: 15
- HTML全文浏览量: 8
- PDF下载量: 0
- 被引次数: 0
