高级搜索

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

脑机接口中脑电图-近红外光谱联合分析进展研究

张力新 周鸿展 王东 孟佳圆 许敏鹏 明东

张力新, 周鸿展, 王东, 孟佳圆, 许敏鹏, 明东. 脑机接口中脑电图-近红外光谱联合分析进展研究[J]. 电子与信息学报, 2024, 46(3): 790-797. doi: 10.11999/JEIT230257
引用本文: 张力新, 周鸿展, 王东, 孟佳圆, 许敏鹏, 明东. 脑机接口中脑电图-近红外光谱联合分析进展研究[J]. 电子与信息学报, 2024, 46(3): 790-797. doi: 10.11999/JEIT230257
ZHANG Lixin, ZHOU Hongzhan, WANG Dong, MENG Jiayuan, XU Minpeng, MING Dong. Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface[J]. Journal of Electronics & Information Technology, 2024, 46(3): 790-797. doi: 10.11999/JEIT230257
Citation: ZHANG Lixin, ZHOU Hongzhan, WANG Dong, MENG Jiayuan, XU Minpeng, MING Dong. Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface[J]. Journal of Electronics & Information Technology, 2024, 46(3): 790-797. doi: 10.11999/JEIT230257

脑机接口中脑电图-近红外光谱联合分析进展研究

doi: 10.11999/JEIT230257
基金项目: 国家重点研发计划(2021YFF1200600),国家自然科学基金(62106173, 81925020),中国博士后科学基金面上项目(2022M712364)
详细信息
    作者简介:

    张力新:男,硕士,研究员,研究方向为生物医学电子学、数字医学影像处理等

    周鸿展:男,硕士生,研究方向为脑机接口中的多模态信号分析

    王东:男,本科生,研究方向为脑机接口中的多模态信号分析

    孟佳圆:女,博士,讲师,研究方向为神经科学与工程,预期、注意等高级认知功能的神经信号特征、机制及其在脑-机接口中的应用

    许敏鹏:男,博士,教授,研究方向为脑-机接口、神经信号处理和神经调控

    明东:男,博士,教授,研究方向为生物医学工程

    通讯作者:

    孟佳圆 mengjiayuan@tju.edu.cn

  • 中图分类号: TN99; R741.044

Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface

Funds: The National Key Research and Development Program of China(2021YFF1200600), The National Natural Science Foundation of China(62106173, 81925020),General Projects of Postdoctoral Science Foundation of China(2022M712364)
  • 摘要: 脑机接口(BCI)能将受试者意图相关的大脑活动转化为外部设备控制指令,在神经疾病治疗、运动康复等方面具有较高应用潜力。BCI的实现需从人脑获取有意义的信号,而脑电图(EEG)可以反映神经电活动,主要用于对反映实时性要求较高的BCI系统;近红外光谱(NIRS)主要反映血流动力学水平,一般用于神经生理状态等需要精确定位脑活跃区域的研究。EEG和NIRS因其非侵入、方便穿戴、成本较低等优点,成为BCI的重要信号获取方法。相比于单模态BCI系统,基于EEG-NIRS 联合分析的混合BCI系统由于具有更丰富的信号特征,在生理状态检测、运动想象等领域得到了越来越多的关注与研究。该文从EEG-NIRS联合分析在脑机接口中应用的研究现状出发,在数据和特征融合程度、层面上归纳最近的相关领域研究现状,并对EEG-NIRS信号处理手段的研究前景进行了展望。
  • 图  1  决策层融合、特征层融合与数据层融合的处理过程示意图

  • [1] BALL T, KERN M, MUTSCHLER I, et al. Signal quality of simultaneously recorded invasive and non-invasive EEG[J]. Neuroimage, 2009, 46(3): 708–716. doi: 10.1016/j.neuroimage.2009.02.028.
    [2] NICOLAS-ALONSO L F and GOMEZ-GIL J. Brain computer interfaces, a review[J]. Sensors, 2012, 12(2): 1211–1279. doi: 10.3390/s120201211.
    [3] HILLMAN E M C. Coupling mechanism and significance of the BOLD signal: A status report[J]. Annual Review of Neuroscience, 2014, 37: 161–181. doi: 10.1146/annurev-neuro-071013-014111.
    [4] BIEßMANN F, PLIS S, MEINECKE F C, et al. Analysis of multimodal neuroimaging data[J]. IEEE Reviews in Biomedical Engineering, 2011, 4: 26–58. doi: 10.1109/RBME.2011.2170675.
    [5] ZUO Cili, JIN Jing, YIN Erwei, et al. Novel hybrid brain-computer interface system based on motor imagery and P300[J]. Cognitive Neurodynamics, 2020, 14(2): 253–265. doi: 10.1007/s11571-019-09560-x.
    [6] FAZLI S, MEHNERT J, STEINBRINK J, et al. Using NIRS as a predictor for EEG-based BCI performance[C]. 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, USA, 2012: 4911–4914.
    [7] TOMITA Y, VIALATTE F B, DREYFUS G, et al. Bimodal BCI using simultaneously NIRS and EEG[J]. IEEE Transactions on Biomedical Engineering, 2014, 61(4): 1274–1284. doi: 10.1109/TBME.2014.2300492.
    [8] KHAN M J, HONG M J, and HONG K S. Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface[J]. Frontiers in Human Neuroscience, 2014, 8: 244. doi: 10.3389/fnhum.2014.00244.
    [9] FAZLI S, MEHNERT J, STEINBRINK J, et al. Enhanced performance by a hybrid NIRS–EEG brain computer interface[J]. Neuroimage, 2012, 59(1): 519–529. doi: 10.1016/j.neuroimage.2011.07.084.
    [10] KWAK Y, SONG W J, and KIM S E. FGANet: fNIRS-guided attention network for hybrid EEG-fNIRS brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 329–339. doi: 10.1109/TNSRE.2022.3149899.
    [11] ALHUDHAIF A. An effective classification framework for brain-computer interface system design based on combining of fNIRS and EEG signals[J]. PeerJ Computer Science, 2021, 7: e537. doi: 10.7717/peerj-cs.537.
    [12] LONG Jinyi, WANG Jue, and YU Tianyou. An efficient framework for EEG analysis with application to hybrid brain computer interfaces based on motor imagery and P300[J]. Computational Intelligence and Neuroscience, 2017, 2017: 9528097. doi: 10.1155/2017/9528097.
    [13] AGHAJANI H, GARBEY M, and OMURTAG A. Measuring mental workload with EEG+fNIRS[J]. Frontiers in Human Neuroscience, 2017, 11: 359. doi: 10.3389/fnhum.2017.00359.
    [14] ALMAJIDY R K, BOUDRIA Y, HOFMANN U G, et al. Multimodal 2D brain computer interface[C]. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 2015: 1067–1070.
    [15] GLOWINSKY S, SAMADANI A, and CHAU T. Limited value of temporo-parietal hemodynamic signals in an optical-electric auditory brain-computer interface[J]. Biomedical Physics & Engineering Express, 2018, 4(4): 045035. doi: 10.1088/2057-1976/aab29a.
    [16] BORGHEAI S B, DELIGANI R J, MCLINDEN J, et al. Multimodal evaluation of mental workload using a hybrid EEG-fNIRS brain-computer interface system[C]. 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), San Francisco, USA, 2019: 973–976.
    [17] KHALIL M A, RAMIREZ M, CAN J, et al. Implementation of machine learning in BCI based lie detection[C]. 2022 IEEE World AI IoT Congress (AIIoT), Seattle, USA, 2022: 213–217.
    [18] SHIN J, KWON J, and IM C H. A ternary hybrid EEG-NIRS brain-computer interface for the classification of brain activation patterns during mental arithmetic, motor imagery, and idle state[J]. Frontiers in Neuroinformatics, 2018, 12: 5. doi: 10.3389/fninf.2018.00005.
    [19] HAN C H, MÜLLER K R, and HWANG H J. Enhanced performance of a brain switch by simultaneous use of EEG and NIRS data for asynhronous brain-computer interface[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(10): 2102–2112. doi: 10.1109/TNSRE.2020.3017167.
    [20] SHIN J, KIM D W, MÜLLER K R, et al. Improvement of information transfer rates using a hybrid EEG-NIRS brain-computer interface with a short trial length: Offline and pseudo-online analyses[J]. Sensors, 2018, 18(6): 1827. doi: 10.3390/s18061827.
    [21] KWON J, SHIN J, and IM C H. Toward a compact hybrid brain-computer interface (BCI): Performance evaluation of multi-class hybrid EEG-fNIRS BCIs with limited number of channels[J]. PLoS One, 2020, 15(3): e0230491. doi: 10.1371/journal.pone.0230491.
    [22] SHIN J, MÜLLER K R, and HWANG H J. Eyes-closed hybrid brain-computer interface employing frontal brain activation[J]. PLoS One, 2018, 13(5): e0196359. doi: 10.1371/journal.pone.0196359.
    [23] WANG Zhongpeng, CAO Cong, ZHOU Yijie, et al. Integrating EEG and NIRS improves BCI performance during motor imagery[C]. 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER), Italy, 2021: 511–514.
    [24] DEHAIS F, DUPRES A, DI FLUMERI G, et al. Monitoring pilot’s cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI[C]. 2018 IEEE International Conference on Systems, Man, and Cybernetics, Miyazaki, Japan, 2018: 544–549.
    [25] QIU Lina, ZHONG Yongshi, XIE Qiuyou, et al. Multi-modal integration of EEG-fNIRS for characterization of brain activity evoked by preferred music[J]. Frontiers in Neurorobotics, 2022, 16: 823435. doi: 10.3389/fnbot.2022.823435.
    [26] CAO Jun, GARRO E M, and ZHAO Yifan. EEG/fNIRS based workload classification using functional brain connectivity and machine learning[J]. Sensors, 2022, 22(19): 7623. doi: 10.3390/s22197623.
    [27] DELIGANI R J, BORGHEAI S B, MCLINDEN J, et al. Multimodal fusion of EEG-fNIRS: A mutual information-based hybrid classification framework[J]. Biomedical Optics Express, 2021, 12(3): 1635–1650. doi: 10.1364/BOE.413666.
    [28] ZHANG Yukun, QIU Shuang, and HE Huiguang. Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion[J]. Journal of Neural Engineering, 2023, 20(2): 026009. doi: 10.1088/1741-2552/acbfdf.
    [29] WANG Yubo, VELUVOLU K C, and LEE M. Time-frequency analysis of band-limited EEG with BMFLC and kalman filter for BCI applications[J]. Journal of Neuroengineering and Rehabilitation, 2013, 10(1): 109. doi: 10.1186/1743-0003-10-109.
    [30] NAGELS-COUNE L, BENITEZ-ANDONEGUI A, REUTER N, et al. Brain-based binary communication using spatiotemporal features of fNIRS responses[J]. Frontiers in Human Neuroscience, 2020, 14: 113. doi: 10.3389/fnhum.2020.00113.
    [31] VON LÜHMANN A, ORTEGA-MARTINEZ A, BOAS D A, et al. Using the general linear model to improve performance in fNIRS single trial analysis and classification: A perspective[J]. Frontiers in Human Neuroscience, 2020, 14: 30. doi: 10.3389/fnhum.2020.00030.
    [32] NAZEER H, NAZEER N, MEHBOOB A, et al. Enhancing classification performance of fNIRS-BCI by identifying cortically active channels using the z-score method[J]. Sensors, 2020, 20(23): 6995. doi: 10.3390/s20236995.
    [33] SHU Xiaokang, YAO Lin, SHENG Xinjun, et al. A hybrid BCI study: Temporal optimization for EEG single-trial classification by exploring hemodynamics from the simultaneously measured NIRS data[C]. 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), Bali, Indonesia, 2014: 914–918.
    [34] KHAN M J and HONG K S. Active brain area identification using EEG-NIRS signal acquisition[C]. 2015 International Conference on Automation, Cognitive Science, Optics, Micro Electro-Mechanical System, and Information Technology (ICACOMIT), Bandung, Indonesia, 2015: 7–11.
    [35] AL-QURAISHI M S, ELAMVAZUTHI I, TANG T B, et al. Bimodal data fusion of simultaneous measurements of EEG and fNIRS during lower limb movements[J]. Brain Sciences, 2021, 11(6): 713. doi: 10.3390/brainsci11060713.
    [36] MENG Ming, DAI Luyang, SHE Qingshan, et al. Crossing time windows optimization based on mutual information for hybrid BCI[J]. Mathematical Biosciences and Engineering, 2021, 18(6): 7919–7935. doi: 10.3934/mbe.2021392.
    [37] LI Rihui, ZHAO Chunli, WANG Chushan, et al. Enhancing fNIRS analysis using EEG rhythmic signatures: An EEG-informed fNIRS analysis study[J]. IEEE Transactions on Biomedical Engineering, 2020, 67(10): 2789–2797. doi: 10.1109/TBME.2020.2971679.
    [38] KHAN M J, GHAFOOR U, and HONG K S. Early detection of hemodynamic responses using EEG: A hybrid EEG-fNIRS study[J]. Frontiers in Human Neuroscience, 2018, 12: 749. doi: 10.3389/fnhum.2018.00479.
    [39] SUN Zhe, HUANG Zihao, DUAN Feng, et al. A novel multimodal approach for hybrid brain–computer interface[J]. IEEE Access, 2020, 8: 89909–89918. doi: 10.1109/ACCESS.2020.2994226.
    [40] NOUR M, ÖZTURK S, and POLAT K. A novel classification framework using multiple bandwidth method with optimized CNN for brain-computer interfaces with EEG-fNIRS signals[J]. Neural Computing & Applications, 2021, 22(33): 15815–15829. doi: 10.1007/s00521-021-06202-4.
    [41] MUGHAL N E, KHAN M J, KHALIL K, et al. EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM[J]. Frontiers in Neurorobotics, 2022, 16: 873239. doi: 10.3389/FNBOT.2022.873239.
    [42] CHEN Jiaming, WANG Dan, HU Bo, et al. MCFHNet: Multi-channel fusion hybrid network for efficient EEG-fNIRS multi-modal motor imagery decoding[C]. 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, UK, 2022: 4821–4825.
    [43] LI Yang, ZHANG Xin, and MING Dong. Early-stage fusion of EEG and fNIRS improves classification of motor imagery[J]. Frontiers in Neuroscience, 2023, 16: 1062889. doi: 10.3389/fnins.2022.1062889.
    [44] GAO Yunyuan, LIU Hongming, FANG Feng, et al. Classification of working memory loads via assessing broken detailed balance of EEG-FNIRS neurovascular coupling measures[J]. IEEE Transactions on Biomedical Engineering, 2023, 70(3): 877–887. doi: 10.1109/TBME.2022.3204718.
    [45] QIU Lina, ZHOU Yongshi, HE Zhipeng, et al. Improved classification performance of EEG-fNIRS multimodal brain-computer interface based on multi-domain features and multi-level progressive learning[J]. Frontiers in Human Neuroscience, 2022, 16: 973959. doi: 10.3389/fnhum.2022.973959.
  • 加载中
图(1)
计量
  • 文章访问数:  410
  • HTML全文浏览量:  202
  • PDF下载量:  86
  • 被引次数: 0
出版历程
  • 收稿日期:  2023-04-12
  • 修回日期:  2023-07-26
  • 网络出版日期:  2023-08-02
  • 刊出日期:  2024-03-27

目录

    /

    返回文章
    返回