基于超材料的电磁神经网络研究进展

马骞; 冯紫瑞; 高欣欣; 顾泽; 游检卫; 崔铁军

doi:10.11999/JEIT231285

基于超材料的电磁神经网络研究进展

doi: 10.11999/JEIT231285

马骞^{1, 2},
冯紫瑞^{1, 2},
高欣欣³,
顾泽^{1, 2},
游检卫^{1, 2},
崔铁军^{1, 2, ,}

1.
东南大学毫米波国家重点实验室南京 210096
2.
东南大学电磁空间研究院南京 210096
3.
香港城市大学太赫兹毫米波国家重点实验室香港 999077

基金项目: 国家自然科学基金(62301147, 62288101, 92167202)，江苏省自然科学基金(BK20230822)，江苏省前沿引领技术基础研究专项(BK20212002)，中央高校基础研究基金(2242023K5002，2242018R30001, 2242022R20017)

详细信息

作者简介:
马骞：男，助理研究员，研究方向为智能超材料、信息超表面和电磁衍射神经网络

冯紫瑞：女，硕士生，研究方向为信息超表面

高欣欣：女，助理研究员，研究方向为人工表面等离激元和智能超材料

顾泽：男，硕士生，研究方向为智能超材料和信息超表面

游检卫：男，教授，研究方向为智能超材料、拓扑超表面和量子超材料

崔铁军：男，教授，中国科学院院士，研究方向为人工电磁等效媒质、人工表面等离激元超材料、信息超材料、智能超表面和计算电磁学等

通讯作者:
崔铁军　tjcui@seu.edu.cn

中图分类号: TN011
计量
- 文章访问数: 1411
- HTML全文浏览量: 483
- PDF下载量: 436
- 被引次数: 0
出版历程
- 收稿日期: 2023-11-20
- 修回日期: 2024-04-30
- 网络出版日期: 2024-05-12
- 刊出日期: 2024-05-30

Research Progress of Electromagnetic Neural Network Based on Metamaterials

MA Qian^{1, 2},
FENG Zirui^{1, 2},
GAO Xinxin³,
GU Ze^{1, 2},
YOU Jianwei^{1, 2},
CUI Tiejun^{1, 2
, ,}

1.
State Key Laboratory of Millimeter Waves, Southeast University, Nanjing 210096, China
2.
Institute of Electromagnetic Space, Southeast University, Nanjing 210096, China
3.
State Key Laboratory of Terahertz and Millimeter Waves, City University of Hong Kong, Hong Kong 999077, China

Funds: The National Natural Science Foundation of China (62301147, 62288101, 92167202), The Natural Science Foundation of Jiangsu Province (BK20230822), The Major Project of Natural Science Foundation of Jiangsu Province (BK20212002), The Fundamental Research Funds for the Central Universities (2242023K5002, 2242018R30001, 2242022R20017)

摘要

摘要: 随着人工智能技术的广泛应用，面向智能计算的算力需求呈井喷式增长。目前芯片的快速发展已经逼近其工艺制程的瓶颈，同时功耗也不断增加，因此高速、高能效的智能计算硬件研究是一个重要方向。以光子电路神经网络和全光衍射神经网络为代表的计算架构因其计算快、功耗低等优势而受到广泛关注。该文回顾了光神经网络的代表性工作，通过3维衍射神经网络和光神经网络芯片化发展两条主线进行介绍，同时，针对光学衍射神经网络和光子神经网络芯片面临的瓶颈和挑战，如网络规模和集成度等，分析比较它们的特点、性能和各自的优劣势。其次，考虑到通用化的发展需求，该文进一步讨论神经形态计算硬件的可编程设计，并在各个部分中介绍了一些可编程神经网络的代表性工作。除了光波段的智能神经网络，本文还讨论了微波衍射神经网络的发展和应用，展示了其可编程能力。最后介绍智能神经形态计算的未来方向和发展趋势，及其在无线通信、信息处理和传感方面的潜在应用。
- 超材料 /
- 全光衍射神经网络 /
- 电磁神经网络 /
- 智能计算
Abstract: With the widespread application of artificial intelligence technology, the demand for computing power for intelligent computing has grown exponentially. At present, the rapid development of chips has approached the bottleneck of its manufacturing process, and power consumption is also increasing. Therefore, research on high-speed, energy-efficient intelligent computing hardware is an important direction. Computing architectures represented by photonic circuit neural networks and all-optical diffraction neural networks have received widespread attention due to their advantages such as fast calculation and low power consumption. This article reviews the representative work of optical neural networks, and introduces it through the two main lines of development of three-dimensional diffractive neural networks and optical neural network chips. At the same time, it focuses on the bottlenecks and challenges faced by optical diffractive neural networks and photonic neural network chips, such as network scale and Integration degree, etc., analyzes and compares their characteristics, performance and respective advantages and disadvantages. Secondly, taking into account the development needs of generalization, this article further discusses the programmable design of neuromorphic computing hardware, and introduces some representative work on programmable neural networks to each part. In addition to intelligent neural networks in the optical band, this article also discusses the development and application of microwave diffraction neural networks and demonstrates their programmability. Finally, the future direction and development trends of intelligent neuromorphic computing are introduced, as well as its potential applications in wireless communications, information processing and sensing.
- Metamaterials /
- Optical diffractive neural network /
- Electromagnetic neural network /
- Intelligent computing.

HTML全文

图 1 全光衍射深度神经网络

下载: 全尺寸图片幻灯片

图 2 衍射光学神经网络和传统电子神经网络对比以及使用衍射神经网络执行分类任务

下载: 全尺寸图片幻灯片

图 3 可重构的光电衍射处理单元

下载: 全尺寸图片幻灯片

图 4 3种不同的神经网络类型以及用于手写数字分类的光电衍射神经网络自适应训练

下载: 全尺寸图片幻灯片

图 5 基于无源超材料的微波衍射神经网络

下载: 全尺寸图片幻灯片

图 6 可编程衍射神经网络

下载: 全尺寸图片幻灯片

图 7 具有可编程权值和非线性激活函数的可编程人工表面等离激元神经网络

下载: 全尺寸图片幻灯片

图 8 基于人工表面等离激元神经网络的无线通信^[60]

下载: 全尺寸图片幻灯片

图 9 使用人工表面等离激元神经网络识别手写数字

下载: 全尺寸图片幻灯片

图 10 平面形式的衍射深度神经网络

下载: 全尺寸图片幻灯片

图 11 复值相干光神经网络^[62]

下载: 全尺寸图片幻灯片

图 12 集成的衍射神经网络和衍射神经元^[62]

下载: 全尺寸图片幻灯片

图 13 多路人脑感知系统及集成在成像传感器芯片上的多路超表面衍射神经网络原理图以及支持超表面的片上多路衍射神经网络对手写数字分类的实验测试^[63]

下载: 全尺寸图片幻灯片

参考文献(66)

[1]	SILVER D, HUANG A, MADDISON C J, et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature, 2016, 529(7587): 484–489. doi: 10.1038/nature16961.
[2]	VAN DIS E A M, BOLLEN J, ZUIDEMA W, et al. ChatGPT: Five priorities for research[J]. Nature, 2023, 614(7947): 224–226. doi: 10.1038/d41586-023-00288-7.
[3]	BROWN T B, MANN B, RYDER N, et al. Language models are few-shot learners[C]. The 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 159.
[4]	XU Xingyuan, REN Guanghui, FELEPPA T, et al. Self-calibrating programmable photonic integrated circuits[J]. Nature Photonics, 2022, 16(8): 595–602. doi: 10.1038/s41566-022-01020-z.
[5]	FELDMANN J, YOUNGBLOOD N, KARPOV M, et al. Parallel convolutional processing using an integrated photonic tensor core[J]. Nature, 2021, 589(7840): 52–58. doi: 10.1038/s41586-020-03070-1.
[6]	FELDMANN J, YOUNGBLOOD N, WRIGHT C D, et al. All-optical spiking neurosynaptic networks with self-learning capabilities[J]. Nature, 2019, 569(7755): 208–214. doi: 10.1038/s41586-019-1157-8.
[7]	WETZSTEIN G, OZCAN A, GIGAN S, et al. Inference in artificial intelligence with deep optics and photonics[J]. Nature, 2020, 588(7836): 39–47. doi: 10.1038/s41586-020-2973-6.
[8]	LI Yuhang, LUO Yi, MENGU D, et al. Quantitative Phase Imaging (QPI) through random diffusers using a diffractive optical network[J]. Light: Advanced Manufacturing, 2023, 4(3): 17. doi: 10.37188/lam.2023.017.
[9]	MENGU D, LUO Yi, RIVENSON Y, et al. Analysis of diffractive optical neural networks and their integration with electronic neural networks[J]. IEEE Journal of Selected Topics in Quantum Electronics, 2020, 26(1): 3700114. doi: 10.1109/JSTQE.2019.2921376.
[10]	WANG Tianyu, SOHONI M M, WRIGHT L G, et al. Image sensing with multilayer nonlinear optical neural networks[J]. Nature Photonics, 2023, 17(5): 408–415. doi: 10.1038/s41566-023-01170-8.
[11]	HUGHES T W, MINKOV M, SHI Yu, et al. Training of photonic neural networks through in situ backpropagation and gradient measurement[J]. Optica, 2018, 5(7): 864–871. doi: 10.1364/optica.5.000864.
[12]	CHEN Yitong, ZHOU Tiankuang, WU Jiamin, et al. Photonic unsupervised learning variational autoencoder for high-throughput and low-latency image transmission[J]. Science Advances, 2023, 9(7): eadf8437. doi: 10.1126/SCIADV.ADF8437.
[13]	LUO Yi, MENGU D, YARDIMCI N T, et al. Design of task-specific optical systems using broadband diffractive neural networks[J]. Light: Science & Applications, 2019, 8: 112. doi: 10.1038/s41377-019-0223-1.
[14]	WILLIAMSON I A D, HUGHES T W, MINKOV M, et al. Reprogrammable electro-optic nonlinear activation functions for optical neural networks[J]. IEEE Journal of Selected Topics in Quantum Electronics, 2020, 26(1): 7700412. doi: 10.1109/jstqe.2019.2930455.
[15]	SHEN Yichen, HARRIS N C, SKIRLO S, et al. Deep learning with coherent nanophotonic circuits[J]. Nature Photonics, 2017, 11(7): 441–446. doi: 10.1038/nphoton.2017.93.
[16]	SHOKRANEH F, GEOFFROY-GAGNON S, NEZAMI M S, et al. A single layer neural network implemented by a 4×4 MZI-Based Optical Processor[J]. IEEE Photonics Journal, 2019, 11(6): 4501612. doi: 10.1109/jphot.2019.2952562.
[17]	POUR FARD M M, WILLIAMSON I A D, EDWARDS M, et al. Experimental realization of arbitrary activation functions for optical neural networks[J]. Optics Express, 2020, 28(8): 12138–12148. doi: 10.1364/OE.391473.
[18]	LIAO Kun, CHEN Ye, YU Zhongcheng, et al. All-optical computing based on convolutional neural networks[J]. Opto-Electronic Advances, 2021, 4(11): 200060. doi: 10.29026/oea.2021.200060.
[19]	HAMERLY R, BERNSTEIN L, SLUDDS A, et al. Large- Scale Optical Neural Network Based on Photoelectric Multiplication[J]. Physical Review X, 2019, 9(2): 021032 doi: 10.1103/PhysRevX.9.021032.
[20]	FU Tingzhao, ZANG Yubin, HUANG Honghao, et al. On-chip photonic diffractive optical neural network based on a spatial domain electromagnetic propagation model[J]. Optics Express, 2021, 29(20): 31924–31940. doi: 10.1364/OE.435183.
[21]	ZAREI S, MARZBAN M R, and KHAVASI A. Integrated photonic neural network based on silicon metalines[J]. Optics Express, 2020, 28(24): 36668–36684. doi: 10.1364/OE.404386.
[22]	FU Tingzhao, ZANG Yubin, HUANG Yuyao, et al. Photonic machine learning with on-chip diffractive optics[J]. Nature Communications, 2023, 14(1): 70. doi: 10.1038/s41467-022-35772-7.
[23]	WANG Zi, CHANG L, WANG Feifan, et al. Integrated photonic metasystem for image classifications at telecommunication wavelength[J]. Nature Communications, 2022, 13(1): 2131. doi: 10.1038/s41467-022-29856-7.
[24]	KHORAM E, CHEN Ang, LIU Dianjing, et al. Nanophotonic media for artificial neural inference[J]. Photonics Research, 2019, 7(8): 823–827. doi: 10.1364/prj.7.000823.
[25]	LIN Xing, RIVENSON Y, YARDIMCI N T, et al. All-optical machine learning using diffractive deep neural networks[J]. Science, 2018, 361(6406): 1004–1008. doi: 10.1126/science.aat8084.
[26]	LI Jingxi, GAN Tianyi, ZHAO Yifan, et al. Unidirectional imaging using deep learning–designed materials[J]. Science Advances, 2023, 9(17): eadg1505. doi: 10.1126/sciadv.adg1505.
[27]	YAN Tao, WU Jiamin, ZHOU Tiankuang, et al. Fourier-space diffractive deep neural network[J]. Physical Review Letters, 2019, 123(2): 023901. doi: 10.1103/PhysRevLett.123.023901.
[28]	RAHMAN M S S, LI Jingxi, MENGU D, et al. Ensemble learning of diffractive optical networks[J]. Light: Science & Applications, 2021, 10(1): 14. doi: 10.1038/s41377-020-00446-w.
[29]	ZHANG Luhe, LI Caiyun, HE Jiangyong, et al. Optical machine learning using time-lens deep neural NetWorks[J]. Photonics, 2021, 8(3): 78. doi: 10.3390/photonics8030078.
[30]	CHANG Julie, SITZMANN V, DUN Xiong, et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification[J]. Scientific Reports, 2018, 8(1): 12324. doi: 10.1038/s41598-018-30619-y.
[31]	QU Geyang, CAI Guiyi, SHA Xinbo, et al. All-dielectric metasurface empowered optical-electronic hybrid neural networks[J]. Laser & Photonics Reviews, 2022, 16(10): 2100732. doi: 10.1002/lpor.202100732.
[32]	XIAO Yongliang, LI Sikun, SITU Guohai, et al. Unitary learning for diffractive deep neural network[J]. Optics and Lasers in Engineering, 2021, 139: 106499. doi: 10.1016/j.optlaseng.2020.106499.
[33]	LI Jingxi, MENGU D, YARDIMCI N T, et al. Spectrally encoded single-pixel machine vision using diffractive networks[J]. Science Advances, 2021, 7(13): eabd7690. doi: 10.1126/sciadv.abd7690.
[34]	LI Jingxi, MENGU D, LUO Yi, et al. Class-specific differential detection in diffractive optical neural networks improves inference accuracy[J]. Advanced Photonics, 2019, 1(4): 046001. doi: 10.1117/1.Ap.1.4.046001.
[35]	ZHOU Tiankuang, LIN Xin, WU Jiamin, et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit[J]. Nature Photonics, 2021, 15(5): 367–373. doi: 10.1038/s41566-021-00796-w.
[36]	CUI Tiejun, QI Meiqing, WAN Xiang, et al. Coding metamaterials, digital metamaterials and programmable metamaterials[J]. Light: Science & Applications, 2014, 3(10): e218. doi: 10.1038/lsa.2014.99.
[37]	CUI Tiejun, LIU Shuo, BAI Guodong, et al. Direct transmission of digital message via programmable coding metasurface[J]. Research, 2019, 2019: 2584509. doi: 10.34133/2019/2584509.
[38]	DAI Junyan, TANG Wankai, CHEN Mingzheng, et al. Wireless communication based on information metasurfaces[J]. IEEE Transactions on Microwave Theory and Techniques, 2021, 69(3): 1493–1510. doi: 10.1109/tmtt.2021.3054662.
[39]	ZHANG Lei, CHEN Mingzheng, TANG Wankai, et al. A wireless communication scheme based on space- and frequency-division multiplexing using digital metasurfaces[J]. Nature Electronics, 2021, 4(3): 218–227. doi: 10.1038/s41928-021-00554-4.
[40]	CHEN Lei, MA Qian, LUO Sisi, et al. Touch-programmable metasurface for various electromagnetic manipulations and encryptions[J]. Small, 2022, 18(45): 2203871. doi: 10.1002/smll.202203871.
[41]	LI Lianlin, SHUANG Ya, MA Qian, et al. Intelligent metasurface imager and recognizer[J]. Light: Science & Applications, 2019, 8: 97. doi: 10.1038/s41377-019-0209-z.
[42]	MA Qian, BAI Guodong, JING Hongbo, et al. Smart metasurface with self-adaptively reprogrammable functions[J]. Light: Science & Applications, 2019, 8: 98. doi: 10.1038/s41377-019-0205-3.
[43]	MA Qian, GAO Wei, XIAO Qiang, et al. Directly wireless communication of human minds via non-invasive brain-computer-metasurface platform[J]. eLight, 2022, 2: 11. doi: 10.1186/s43593-022-00019-x.
[44]	GAO Xinxin, MA Qian, GU Ze, et al. Programmable surface plasmonic neural networks for microwave detection and processing[J]. Nature Electronics, 2023, 6(4): 319–328. doi: 10.1038/s41928-023-00951-x.
[45]	LIU Che, MA Qian, LUO Zhangjie, et al. A programmable diffractive deep neural network based on a digital-coding metasurface array[J]. Nature Electronics, 2022, 5(2): 113–122. doi: 10.1038/s41928-022-00719-9.
[46]	QIAN Chao, WANG Zhedong, QIAN Haoliang, et al. Dynamic recognition and mirage using neuro-metamaterials[J]. Nature Communications, 2022, 13(1): 2694. doi: 10.1038/s41467-022-30377-6.
[47]	GAO Xinxin, MA Qian, GU Ze, et al. Reconfigurable spoof localized surface plasmonic for frequency detections[J]. Laser & Photonics Reviews, 2023, 17(11): 2300267. doi: 10.1002/lpor.202300267.
[48]	GAO Xinxin, CUI Wenyi, GU Ze, et al. Multimode and reconfigurable phase shifter of spoof surface plasmons[J]. IEEE Transactions on Antennas and Propagation, 2023, 71(6): 5361–5369. doi: 10.1109/tap.2023.3262348.
[49]	GAO Xinxin, CHEN Baojie, SHUM K M, et al. Multifunctional terahertz spoof plasmonic devices[J]. Advanced Materials Technologies, 2023, 8(12): 2202050. doi: 10.1002/admt.202202050.
[50]	GAO Xinxin, GU Ze, MA Qian, et al. Reprogrammable spoof plasmonic modulator[J]. Advanced Functional Materials, 2023, 33(18): 2212328. doi: 10.1002/adfm.202212328.
[51]	GAO Xinxin, ZHANG Haochi, WU Liangwei, et al. Programmable multifunctional device based on spoof surface Plasmon polaritons[J]. IEEE Transactions on Antennas and Propagation, 2020, 68(5): 3770–3779. doi: 10.1109/tap.2020.2969745.
[52]	GAO Xinxin, ZHANG Jingjing, MA Qian, et al. Nonmagnetic spoof plasmonic isolator based on parametric amplification[J]. Laser & Photonics Reviews, 2022, 16(4): 2100578. doi: 10.1002/lpor.202100578.
[53]	GAO Xinxin, ZHANG Jingjing, LUO Yu, et al. Reconfigurable parametric amplifications of spoof surface plasmons[J]. Advanced Science, 2021, 8(17): 2100795. doi: 10.1002/advs.202100795.
[54]	GAO Xinxin, ZHANG Jingjing, ZHANG Haochi, et al. Dynamic controls of second‐harmonic generations in both forward and backward modes using reconfigurable plasmonic metawaveguide[J]. Advanced Optical Materials, 2020, 8(8): 1902058. doi: 10.1002/adom.201902058.
[55]	GAO Xinxin, ZHANG Haochi, HE Peihang, et al. Crosstalk suppression based on mode mismatch between spoof SPP transmission line and microstrip[J]. IEEE Transactions on Components, Packaging and Manufacturing Technology, 2019, 9(11): 2267–2275. doi: 10.1109/tcpmt.2019.2931373.
[56]	ZHANG Haochi, ZHANG Lepeng, HE Peihang, et al. A plasmonic route for the integrated wireless communication of subdiffraction-limited signals[J]. Light: Science & Applications, 2020, 9: 113. doi: 10.1038/s41377-020-00355-y.
[57]	JOY S R, EREMENTCHOUK M, YU Hao, et al. Spoof Plasmon interconnects—communications beyond RC limit[J]. IEEE Transactions on Communications, 2019, 67(1): 599–610. doi: 10.1109/tcomm.2018.2874242.
[58]	GOI E, SCHOENHARDT S, and GU Ming. Direct retrieval of Zernike-based pupil functions using integrated diffractive deep neural networks[J]. Nature Communications, 2022, 13(1): 7531. doi: 10.1038/s41467-022-35349-4.
[59]	QIAN Chao, LIN Xiao, LIN Xiaobin, et al. Performing optical logic operations by a diffractive neural network[J]. Light: Science & Applications, 2020, 9: 59. doi: 10.1038/s41377-020-0303-2.
[60]	GAO Xinxin and CUI Tiejun. Using surface plasmons to create programmable neural networks[J]. Nature Electronics, 2023, 6(4): 266–267. doi: 10.1038/s41928-023-00952-w.
[61]	ZHANG H, GU M, JIANG X D, et al. An optical neural chip for implementing complex-valued neural network[J]. Nature Communications, 2021, 12(1): 457. doi: 10.1038/s41467-020-20719-7.
[62]	ZHU H H, ZOU J, ZHANG H, et al. Space-efficient optical computing with an integrated chip diffractive neural network[J]. Nature Communications, 2022, 13(1): 1044. doi: 10.1038/s41467-022-28702-0.
[63]	LUO Xuhao, HU Yueqiang, OU Xiangnian et al. Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible[J]. Light: Science & Applications, 2022, 11(1): 158. doi: 10.1038/s41377-022-00844-2.
[64]	HUANG Zebin, WANG Peipei, LIU Junmin, et al. All-optical signal processing of vortex beams with diffractive deep neural networks[J]. Physical Review Applied, 2021, 15(1): 014037. doi: 10.1103/PhysRevApplied.15.014037.
[65]	WANG Zhedong, QIAN Chao, FAN Zhixiang, et al. Arbitrary polarization readout with dual-channel neuro-metasurfaces[J]. Advanced Science, 2023, 10(5): 2204699. doi: 10.1002/advs.202204699.
[66]	MA Qian, GAO Xinxin, GU Ze, et al. Intelligent neuromorphic computing based on nanophotonics and metamaterials[J]. MRS Communications. doi: 10.1557/s43579-024-00520-z.