脑机接口中脑电图-近红外光谱联合分析进展研究

张力新; 周鸿展; 王东; 孟佳圆; 许敏鹏; 明东

doi:10.11999/JEIT230257

脑机接口中脑电图-近红外光谱联合分析进展研究

doi: 10.11999/JEIT230257

张力新¹,
周鸿展¹,
王东¹,
孟佳圆^{1, 2, ,},
许敏鹏^{1, 2},
明东^{1, 2}

1.
天津大学医学工程与转化医学研究院天津 300072
2.
天津大学精密仪器与光电子工程学院天津 300072

基金项目: 国家重点研发计划(2021YFF1200600)，国家自然科学基金(62106173, 81925020)，中国博士后科学基金面上项目(2022M712364)

详细信息

作者简介:
张力新：男，硕士，研究员，研究方向为生物医学电子学、数字医学影像处理等

周鸿展：男，硕士生，研究方向为脑机接口中的多模态信号分析

王东：男，本科生，研究方向为脑机接口中的多模态信号分析

孟佳圆：女，博士，讲师，研究方向为神经科学与工程，预期、注意等高级认知功能的神经信号特征、机制及其在脑-机接口中的应用

许敏鹏：男，博士，教授，研究方向为脑-机接口、神经信号处理和神经调控

明东：男，博士，教授，研究方向为生物医学工程

通讯作者:
孟佳圆　mengjiayuan@tju.edu.cn

中图分类号: TN99; R741.044
计量
- 文章访问数: 459
- HTML全文浏览量: 207
- PDF下载量: 101
- 被引次数: 0
出版历程
- 收稿日期: 2023-04-12
- 修回日期: 2023-07-26
- 网络出版日期: 2023-08-02
- 刊出日期: 2024-03-27

Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface

ZHANG Lixin¹,
ZHOU Hongzhan¹,
WANG Dong¹,
MENG Jiayuan^{1, 2
, ,},
XU Minpeng^{1, 2},
MING Dong^{1, 2}

1.
Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China
2.
Precision Instruments & Optoelectronics Engineering, Tianjin University, Tianjin 300072, China

Funds: The National Key Research and Development Program of China(2021YFF1200600), The National Natural Science Foundation of China(62106173, 81925020),General Projects of Postdoctoral Science Foundation of China(2022M712364)

摘要

摘要: 脑机接口(BCI)能将受试者意图相关的大脑活动转化为外部设备控制指令，在神经疾病治疗、运动康复等方面具有较高应用潜力。BCI的实现需从人脑获取有意义的信号，而脑电图(EEG)可以反映神经电活动，主要用于对反映实时性要求较高的BCI系统；近红外光谱(NIRS)主要反映血流动力学水平，一般用于神经生理状态等需要精确定位脑活跃区域的研究。EEG和NIRS因其非侵入、方便穿戴、成本较低等优点，成为BCI的重要信号获取方法。相比于单模态BCI系统，基于EEG-NIRS 联合分析的混合BCI系统由于具有更丰富的信号特征，在生理状态检测、运动想象等领域得到了越来越多的关注与研究。该文从EEG-NIRS联合分析在脑机接口中应用的研究现状出发，在数据和特征融合程度、层面上归纳最近的相关领域研究现状，并对EEG-NIRS信号处理手段的研究前景进行了展望。
- 信号处理 /
- 脑机接口 /
- 脑电图 /
- 近红外光谱
Abstract: Brain-Computer Interface (BCI) can convert the brain activity related to the subject's intention into external device control instructions, which have high application potential in treating neurological diseases, motor rehabilitation, and other aspects. Considering that the materialization of BCI needs to obtain meaningful signals from the human brain, ElectroEncephaloGraphy (EEG) and Near-InfRared Spectroscopy (NIRS) has become important signal acquisition methods for BCI because they are non-invasive, convenient to wear, and relatively cheap. EEG reflects neural electrical activity and is widely applied in BCI systems with high real-time response requirements; NIRS mainly reflects the level of hemodynamics and is mainly utilized in research with precise localization of active brain regions, such as identifying neurophysiological status. Compared with the single-mode BCI system, the BCI system based on EEG-NIRS combined analysis has attracted interest and research in physiological state detection, motor imagination, etc., because of its richer signal characteristics. This review begins with the application of EEG-NIRS combined data analysis in BCI, summarizes the current development on the data and feature fusion level, and looks forward to the research prospects of EEG-NIRS signal processing methods.
- Signal process /
- Brain-Computer Interface (BCI) /
- ElectroEncephaloGraphy (EEG) /
- Near-InfRared Spectroscopy (NIRS)

HTML全文

图 1 决策层融合、特征层融合与数据层融合的处理过程示意图

下载: 全尺寸图片幻灯片

参考文献(45)

[1]	BALL T, KERN M, MUTSCHLER I, et al. Signal quality of simultaneously recorded invasive and non-invasive EEG[J]. Neuroimage, 2009, 46(3): 708–716. doi: 10.1016/j.neuroimage.2009.02.028.
[2]	NICOLAS-ALONSO L F and GOMEZ-GIL J. Brain computer interfaces, a review[J]. Sensors, 2012, 12(2): 1211–1279. doi: 10.3390/s120201211.
[3]	HILLMAN E M C. Coupling mechanism and significance of the BOLD signal: A status report[J]. Annual Review of Neuroscience, 2014, 37: 161–181. doi: 10.1146/annurev-neuro-071013-014111.
[4]	BIEßMANN F, PLIS S, MEINECKE F C, et al. Analysis of multimodal neuroimaging data[J]. IEEE Reviews in Biomedical Engineering, 2011, 4: 26–58. doi: 10.1109/RBME.2011.2170675.
[5]	ZUO Cili, JIN Jing, YIN Erwei, et al. Novel hybrid brain-computer interface system based on motor imagery and P300[J]. Cognitive Neurodynamics, 2020, 14(2): 253–265. doi: 10.1007/s11571-019-09560-x.
[6]	FAZLI S, MEHNERT J, STEINBRINK J, et al. Using NIRS as a predictor for EEG-based BCI performance[C]. 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, USA, 2012: 4911–4914.
[7]	TOMITA Y, VIALATTE F B, DREYFUS G, et al. Bimodal BCI using simultaneously NIRS and EEG[J]. IEEE Transactions on Biomedical Engineering, 2014, 61(4): 1274–1284. doi: 10.1109/TBME.2014.2300492.
[8]	KHAN M J, HONG M J, and HONG K S. Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface[J]. Frontiers in Human Neuroscience, 2014, 8: 244. doi: 10.3389/fnhum.2014.00244.
[9]	FAZLI S, MEHNERT J, STEINBRINK J, et al. Enhanced performance by a hybrid NIRS–EEG brain computer interface[J]. Neuroimage, 2012, 59(1): 519–529. doi: 10.1016/j.neuroimage.2011.07.084.
[10]	KWAK Y, SONG W J, and KIM S E. FGANet: fNIRS-guided attention network for hybrid EEG-fNIRS brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 329–339. doi: 10.1109/TNSRE.2022.3149899.
[11]	ALHUDHAIF A. An effective classification framework for brain-computer interface system design based on combining of fNIRS and EEG signals[J]. PeerJ Computer Science, 2021, 7: e537. doi: 10.7717/peerj-cs.537.
[12]	LONG Jinyi, WANG Jue, and YU Tianyou. An efficient framework for EEG analysis with application to hybrid brain computer interfaces based on motor imagery and P300[J]. Computational Intelligence and Neuroscience, 2017, 2017: 9528097. doi: 10.1155/2017/9528097.
[13]	AGHAJANI H, GARBEY M, and OMURTAG A. Measuring mental workload with EEG+fNIRS[J]. Frontiers in Human Neuroscience, 2017, 11: 359. doi: 10.3389/fnhum.2017.00359.
[14]	ALMAJIDY R K, BOUDRIA Y, HOFMANN U G, et al. Multimodal 2D brain computer interface[C]. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 2015: 1067–1070.
[15]	GLOWINSKY S, SAMADANI A, and CHAU T. Limited value of temporo-parietal hemodynamic signals in an optical-electric auditory brain-computer interface[J]. Biomedical Physics & Engineering Express, 2018, 4(4): 045035. doi: 10.1088/2057-1976/aab29a.
[16]	BORGHEAI S B, DELIGANI R J, MCLINDEN J, et al. Multimodal evaluation of mental workload using a hybrid EEG-fNIRS brain-computer interface system[C]. 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), San Francisco, USA, 2019: 973–976.
[17]	KHALIL M A, RAMIREZ M, CAN J, et al. Implementation of machine learning in BCI based lie detection[C]. 2022 IEEE World AI IoT Congress (AIIoT), Seattle, USA, 2022: 213–217.
[18]	SHIN J, KWON J, and IM C H. A ternary hybrid EEG-NIRS brain-computer interface for the classification of brain activation patterns during mental arithmetic, motor imagery, and idle state[J]. Frontiers in Neuroinformatics, 2018, 12: 5. doi: 10.3389/fninf.2018.00005.
[19]	HAN C H, MÜLLER K R, and HWANG H J. Enhanced performance of a brain switch by simultaneous use of EEG and NIRS data for asynhronous brain-computer interface[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(10): 2102–2112. doi: 10.1109/TNSRE.2020.3017167.
[20]	SHIN J, KIM D W, MÜLLER K R, et al. Improvement of information transfer rates using a hybrid EEG-NIRS brain-computer interface with a short trial length: Offline and pseudo-online analyses[J]. Sensors, 2018, 18(6): 1827. doi: 10.3390/s18061827.
[21]	KWON J, SHIN J, and IM C H. Toward a compact hybrid brain-computer interface (BCI): Performance evaluation of multi-class hybrid EEG-fNIRS BCIs with limited number of channels[J]. PLoS One, 2020, 15(3): e0230491. doi: 10.1371/journal.pone.0230491.
[22]	SHIN J, MÜLLER K R, and HWANG H J. Eyes-closed hybrid brain-computer interface employing frontal brain activation[J]. PLoS One, 2018, 13(5): e0196359. doi: 10.1371/journal.pone.0196359.
[23]	WANG Zhongpeng, CAO Cong, ZHOU Yijie, et al. Integrating EEG and NIRS improves BCI performance during motor imagery[C]. 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER), Italy, 2021: 511–514.
[24]	DEHAIS F, DUPRES A, DI FLUMERI G, et al. Monitoring pilot’s cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI[C]. 2018 IEEE International Conference on Systems, Man, and Cybernetics, Miyazaki, Japan, 2018: 544–549.
[25]	QIU Lina, ZHONG Yongshi, XIE Qiuyou, et al. Multi-modal integration of EEG-fNIRS for characterization of brain activity evoked by preferred music[J]. Frontiers in Neurorobotics, 2022, 16: 823435. doi: 10.3389/fnbot.2022.823435.
[26]	CAO Jun, GARRO E M, and ZHAO Yifan. EEG/fNIRS based workload classification using functional brain connectivity and machine learning[J]. Sensors, 2022, 22(19): 7623. doi: 10.3390/s22197623.
[27]	DELIGANI R J, BORGHEAI S B, MCLINDEN J, et al. Multimodal fusion of EEG-fNIRS: A mutual information-based hybrid classification framework[J]. Biomedical Optics Express, 2021, 12(3): 1635–1650. doi: 10.1364/BOE.413666.
[28]	ZHANG Yukun, QIU Shuang, and HE Huiguang. Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion[J]. Journal of Neural Engineering, 2023, 20(2): 026009. doi: 10.1088/1741-2552/acbfdf.
[29]	WANG Yubo, VELUVOLU K C, and LEE M. Time-frequency analysis of band-limited EEG with BMFLC and kalman filter for BCI applications[J]. Journal of Neuroengineering and Rehabilitation, 2013, 10(1): 109. doi: 10.1186/1743-0003-10-109.
[30]	NAGELS-COUNE L, BENITEZ-ANDONEGUI A, REUTER N, et al. Brain-based binary communication using spatiotemporal features of fNIRS responses[J]. Frontiers in Human Neuroscience, 2020, 14: 113. doi: 10.3389/fnhum.2020.00113.
[31]	VON LÜHMANN A, ORTEGA-MARTINEZ A, BOAS D A, et al. Using the general linear model to improve performance in fNIRS single trial analysis and classification: A perspective[J]. Frontiers in Human Neuroscience, 2020, 14: 30. doi: 10.3389/fnhum.2020.00030.
[32]	NAZEER H, NAZEER N, MEHBOOB A, et al. Enhancing classification performance of fNIRS-BCI by identifying cortically active channels using the z-score method[J]. Sensors, 2020, 20(23): 6995. doi: 10.3390/s20236995.
[33]	SHU Xiaokang, YAO Lin, SHENG Xinjun, et al. A hybrid BCI study: Temporal optimization for EEG single-trial classification by exploring hemodynamics from the simultaneously measured NIRS data[C]. 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), Bali, Indonesia, 2014: 914–918.
[34]	KHAN M J and HONG K S. Active brain area identification using EEG-NIRS signal acquisition[C]. 2015 International Conference on Automation, Cognitive Science, Optics, Micro Electro-Mechanical System, and Information Technology (ICACOMIT), Bandung, Indonesia, 2015: 7–11.
[35]	AL-QURAISHI M S, ELAMVAZUTHI I, TANG T B, et al. Bimodal data fusion of simultaneous measurements of EEG and fNIRS during lower limb movements[J]. Brain Sciences, 2021, 11(6): 713. doi: 10.3390/brainsci11060713.
[36]	MENG Ming, DAI Luyang, SHE Qingshan, et al. Crossing time windows optimization based on mutual information for hybrid BCI[J]. Mathematical Biosciences and Engineering, 2021, 18(6): 7919–7935. doi: 10.3934/mbe.2021392.
[37]	LI Rihui, ZHAO Chunli, WANG Chushan, et al. Enhancing fNIRS analysis using EEG rhythmic signatures: An EEG-informed fNIRS analysis study[J]. IEEE Transactions on Biomedical Engineering, 2020, 67(10): 2789–2797. doi: 10.1109/TBME.2020.2971679.
[38]	KHAN M J, GHAFOOR U, and HONG K S. Early detection of hemodynamic responses using EEG: A hybrid EEG-fNIRS study[J]. Frontiers in Human Neuroscience, 2018, 12: 749. doi: 10.3389/fnhum.2018.00479.
[39]	SUN Zhe, HUANG Zihao, DUAN Feng, et al. A novel multimodal approach for hybrid brain–computer interface[J]. IEEE Access, 2020, 8: 89909–89918. doi: 10.1109/ACCESS.2020.2994226.
[40]	NOUR M, ÖZTURK S, and POLAT K. A novel classification framework using multiple bandwidth method with optimized CNN for brain-computer interfaces with EEG-fNIRS signals[J]. Neural Computing & Applications, 2021, 22(33): 15815–15829. doi: 10.1007/s00521-021-06202-4.
[41]	MUGHAL N E, KHAN M J, KHALIL K, et al. EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM[J]. Frontiers in Neurorobotics, 2022, 16: 873239. doi: 10.3389/FNBOT.2022.873239.
[42]	CHEN Jiaming, WANG Dan, HU Bo, et al. MCFHNet: Multi-channel fusion hybrid network for efficient EEG-fNIRS multi-modal motor imagery decoding[C]. 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, UK, 2022: 4821–4825.
[43]	LI Yang, ZHANG Xin, and MING Dong. Early-stage fusion of EEG and fNIRS improves classification of motor imagery[J]. Frontiers in Neuroscience, 2023, 16: 1062889. doi: 10.3389/fnins.2022.1062889.
[44]	GAO Yunyuan, LIU Hongming, FANG Feng, et al. Classification of working memory loads via assessing broken detailed balance of EEG-FNIRS neurovascular coupling measures[J]. IEEE Transactions on Biomedical Engineering, 2023, 70(3): 877–887. doi: 10.1109/TBME.2022.3204718.
[45]	QIU Lina, ZHOU Yongshi, HE Zhipeng, et al. Improved classification performance of EEG-fNIRS multimodal brain-computer interface based on multi-domain features and multi-level progressive learning[J]. Frontiers in Human Neuroscience, 2022, 16: 973959. doi: 10.3389/fnhum.2022.973959.