Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface

ZHANG Lixin; ZHOU Hongzhan; WANG Dong; MENG Jiayuan; XU Minpeng; MING Dong

doi:10.11999/JEIT230257

Volume 46 Issue 3

Mar. 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2024 > 46(3): 790-797

ZHANG Lixin, ZHOU Hongzhan, WANG Dong, MENG Jiayuan, XU Minpeng, MING Dong. Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface[J]. Journal of Electronics & Information Technology, 2024, 46(3): 790-797. doi: 10.11999/JEIT230257

Citation:

ZHANG Lixin, ZHOU Hongzhan, WANG Dong, MENG Jiayuan, XU Minpeng, MING Dong. Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface[J]. Journal of Electronics & Information Technology, 2024, 46(3): 790-797. doi: 10.11999/JEIT230257

Citation:

ZHANG Lixin, ZHOU Hongzhan, WANG Dong, MENG Jiayuan, XU Minpeng, MING Dong. Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface[J]. Journal of Electronics & Information Technology, 2024, 46(3): 790-797. doi: 10.11999/JEIT230257

PDF( 525 KB)

Research Progress of ElectroEncephaloGraphy-Near-InfRared Spectroscopy Combined Analysis in Brain-Computer Interface

doi: 10.11999/JEIT230257 cstr: 32379.14.JEIT230257

ZHANG Lixin¹,
ZHOU Hongzhan¹,
WANG Dong¹,
MENG Jiayuan^{1, 2
,
,},
XU Minpeng^{1, 2},
MING Dong^{1, 2}

1.
Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China
2.
Precision Instruments & Optoelectronics Engineering, Tianjin University, Tianjin 300072, China

Funds: The National Key Research and Development Program of China(2021YFF1200600), The National Natural Science Foundation of China(62106173, 81925020),General Projects of Postdoctoral Science Foundation of China(2022M712364)

Received Date: 2023-04-12
Rev Recd Date: 2023-07-26

Available Online: 2023-08-02

Publish Date: 2024-03-27

Abstract

Abstract

Brain-Computer Interface (BCI) can convert the brain activity related to the subject's intention into external device control instructions, which have high application potential in treating neurological diseases, motor rehabilitation, and other aspects. Considering that the materialization of BCI needs to obtain meaningful signals from the human brain, ElectroEncephaloGraphy (EEG) and Near-InfRared Spectroscopy (NIRS) has become important signal acquisition methods for BCI because they are non-invasive, convenient to wear, and relatively cheap. EEG reflects neural electrical activity and is widely applied in BCI systems with high real-time response requirements; NIRS mainly reflects the level of hemodynamics and is mainly utilized in research with precise localization of active brain regions, such as identifying neurophysiological status. Compared with the single-mode BCI system, the BCI system based on EEG-NIRS combined analysis has attracted interest and research in physiological state detection, motor imagination, etc., because of its richer signal characteristics. This review begins with the application of EEG-NIRS combined data analysis in BCI, summarizes the current development on the data and feature fusion level, and looks forward to the research prospects of EEG-NIRS signal processing methods.
- Signal process,
- Brain-Computer Interface (BCI),
- ElectroEncephaloGraphy (EEG),
- Near-InfRared Spectroscopy (NIRS)

FullText(HTML)

References(45)

References

[1]	BALL T, KERN M, MUTSCHLER I, et al. Signal quality of simultaneously recorded invasive and non-invasive EEG[J]. Neuroimage, 2009, 46(3): 708–716. doi: 10.1016/j.neuroimage.2009.02.028.
[2]	NICOLAS-ALONSO L F and GOMEZ-GIL J. Brain computer interfaces, a review[J]. Sensors, 2012, 12(2): 1211–1279. doi: 10.3390/s120201211.
[3]	HILLMAN E M C. Coupling mechanism and significance of the BOLD signal: A status report[J]. Annual Review of Neuroscience, 2014, 37: 161–181. doi: 10.1146/annurev-neuro-071013-014111.
[4]	BIEßMANN F, PLIS S, MEINECKE F C, et al. Analysis of multimodal neuroimaging data[J]. IEEE Reviews in Biomedical Engineering, 2011, 4: 26–58. doi: 10.1109/RBME.2011.2170675.
[5]	ZUO Cili, JIN Jing, YIN Erwei, et al. Novel hybrid brain-computer interface system based on motor imagery and P300[J]. Cognitive Neurodynamics, 2020, 14(2): 253–265. doi: 10.1007/s11571-019-09560-x.
[6]	FAZLI S, MEHNERT J, STEINBRINK J, et al. Using NIRS as a predictor for EEG-based BCI performance[C]. 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, USA, 2012: 4911–4914.
[7]	TOMITA Y, VIALATTE F B, DREYFUS G, et al. Bimodal BCI using simultaneously NIRS and EEG[J]. IEEE Transactions on Biomedical Engineering, 2014, 61(4): 1274–1284. doi: 10.1109/TBME.2014.2300492.
[8]	KHAN M J, HONG M J, and HONG K S. Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface[J]. Frontiers in Human Neuroscience, 2014, 8: 244. doi: 10.3389/fnhum.2014.00244.
[9]	FAZLI S, MEHNERT J, STEINBRINK J, et al. Enhanced performance by a hybrid NIRS–EEG brain computer interface[J]. Neuroimage, 2012, 59(1): 519–529. doi: 10.1016/j.neuroimage.2011.07.084.
[10]	KWAK Y, SONG W J, and KIM S E. FGANet: fNIRS-guided attention network for hybrid EEG-fNIRS brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 329–339. doi: 10.1109/TNSRE.2022.3149899.
[11]	ALHUDHAIF A. An effective classification framework for brain-computer interface system design based on combining of fNIRS and EEG signals[J]. PeerJ Computer Science, 2021, 7: e537. doi: 10.7717/peerj-cs.537.
[12]	LONG Jinyi, WANG Jue, and YU Tianyou. An efficient framework for EEG analysis with application to hybrid brain computer interfaces based on motor imagery and P300[J]. Computational Intelligence and Neuroscience, 2017, 2017: 9528097. doi: 10.1155/2017/9528097.
[13]	AGHAJANI H, GARBEY M, and OMURTAG A. Measuring mental workload with EEG+fNIRS[J]. Frontiers in Human Neuroscience, 2017, 11: 359. doi: 10.3389/fnhum.2017.00359.
[14]	ALMAJIDY R K, BOUDRIA Y, HOFMANN U G, et al. Multimodal 2D brain computer interface[C]. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 2015: 1067–1070.
[15]	GLOWINSKY S, SAMADANI A, and CHAU T. Limited value of temporo-parietal hemodynamic signals in an optical-electric auditory brain-computer interface[J]. Biomedical Physics & Engineering Express, 2018, 4(4): 045035. doi: 10.1088/2057-1976/aab29a.
[16]	BORGHEAI S B, DELIGANI R J, MCLINDEN J, et al. Multimodal evaluation of mental workload using a hybrid EEG-fNIRS brain-computer interface system[C]. 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), San Francisco, USA, 2019: 973–976.
[17]	KHALIL M A, RAMIREZ M, CAN J, et al. Implementation of machine learning in BCI based lie detection[C]. 2022 IEEE World AI IoT Congress (AIIoT), Seattle, USA, 2022: 213–217.
[18]	SHIN J, KWON J, and IM C H. A ternary hybrid EEG-NIRS brain-computer interface for the classification of brain activation patterns during mental arithmetic, motor imagery, and idle state[J]. Frontiers in Neuroinformatics, 2018, 12: 5. doi: 10.3389/fninf.2018.00005.
[19]	HAN C H, MÜLLER K R, and HWANG H J. Enhanced performance of a brain switch by simultaneous use of EEG and NIRS data for asynhronous brain-computer interface[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(10): 2102–2112. doi: 10.1109/TNSRE.2020.3017167.
[20]	SHIN J, KIM D W, MÜLLER K R, et al. Improvement of information transfer rates using a hybrid EEG-NIRS brain-computer interface with a short trial length: Offline and pseudo-online analyses[J]. Sensors, 2018, 18(6): 1827. doi: 10.3390/s18061827.
[21]	KWON J, SHIN J, and IM C H. Toward a compact hybrid brain-computer interface (BCI): Performance evaluation of multi-class hybrid EEG-fNIRS BCIs with limited number of channels[J]. PLoS One, 2020, 15(3): e0230491. doi: 10.1371/journal.pone.0230491.
[22]	SHIN J, MÜLLER K R, and HWANG H J. Eyes-closed hybrid brain-computer interface employing frontal brain activation[J]. PLoS One, 2018, 13(5): e0196359. doi: 10.1371/journal.pone.0196359.
[23]	WANG Zhongpeng, CAO Cong, ZHOU Yijie, et al. Integrating EEG and NIRS improves BCI performance during motor imagery[C]. 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER), Italy, 2021: 511–514.
[24]	DEHAIS F, DUPRES A, DI FLUMERI G, et al. Monitoring pilot’s cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI[C]. 2018 IEEE International Conference on Systems, Man, and Cybernetics, Miyazaki, Japan, 2018: 544–549.
[25]	QIU Lina, ZHONG Yongshi, XIE Qiuyou, et al. Multi-modal integration of EEG-fNIRS for characterization of brain activity evoked by preferred music[J]. Frontiers in Neurorobotics, 2022, 16: 823435. doi: 10.3389/fnbot.2022.823435.
[26]	CAO Jun, GARRO E M, and ZHAO Yifan. EEG/fNIRS based workload classification using functional brain connectivity and machine learning[J]. Sensors, 2022, 22(19): 7623. doi: 10.3390/s22197623.
[27]	DELIGANI R J, BORGHEAI S B, MCLINDEN J, et al. Multimodal fusion of EEG-fNIRS: A mutual information-based hybrid classification framework[J]. Biomedical Optics Express, 2021, 12(3): 1635–1650. doi: 10.1364/BOE.413666.
[28]	ZHANG Yukun, QIU Shuang, and HE Huiguang. Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion[J]. Journal of Neural Engineering, 2023, 20(2): 026009. doi: 10.1088/1741-2552/acbfdf.
[29]	WANG Yubo, VELUVOLU K C, and LEE M. Time-frequency analysis of band-limited EEG with BMFLC and kalman filter for BCI applications[J]. Journal of Neuroengineering and Rehabilitation, 2013, 10(1): 109. doi: 10.1186/1743-0003-10-109.
[30]	NAGELS-COUNE L, BENITEZ-ANDONEGUI A, REUTER N, et al. Brain-based binary communication using spatiotemporal features of fNIRS responses[J]. Frontiers in Human Neuroscience, 2020, 14: 113. doi: 10.3389/fnhum.2020.00113.
[31]	VON LÜHMANN A, ORTEGA-MARTINEZ A, BOAS D A, et al. Using the general linear model to improve performance in fNIRS single trial analysis and classification: A perspective[J]. Frontiers in Human Neuroscience, 2020, 14: 30. doi: 10.3389/fnhum.2020.00030.
[32]	NAZEER H, NAZEER N, MEHBOOB A, et al. Enhancing classification performance of fNIRS-BCI by identifying cortically active channels using the z-score method[J]. Sensors, 2020, 20(23): 6995. doi: 10.3390/s20236995.
[33]	SHU Xiaokang, YAO Lin, SHENG Xinjun, et al. A hybrid BCI study: Temporal optimization for EEG single-trial classification by exploring hemodynamics from the simultaneously measured NIRS data[C]. 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), Bali, Indonesia, 2014: 914–918.
[34]	KHAN M J and HONG K S. Active brain area identification using EEG-NIRS signal acquisition[C]. 2015 International Conference on Automation, Cognitive Science, Optics, Micro Electro-Mechanical System, and Information Technology (ICACOMIT), Bandung, Indonesia, 2015: 7–11.
[35]	AL-QURAISHI M S, ELAMVAZUTHI I, TANG T B, et al. Bimodal data fusion of simultaneous measurements of EEG and fNIRS during lower limb movements[J]. Brain Sciences, 2021, 11(6): 713. doi: 10.3390/brainsci11060713.
[36]	MENG Ming, DAI Luyang, SHE Qingshan, et al. Crossing time windows optimization based on mutual information for hybrid BCI[J]. Mathematical Biosciences and Engineering, 2021, 18(6): 7919–7935. doi: 10.3934/mbe.2021392.
[37]	LI Rihui, ZHAO Chunli, WANG Chushan, et al. Enhancing fNIRS analysis using EEG rhythmic signatures: An EEG-informed fNIRS analysis study[J]. IEEE Transactions on Biomedical Engineering, 2020, 67(10): 2789–2797. doi: 10.1109/TBME.2020.2971679.
[38]	KHAN M J, GHAFOOR U, and HONG K S. Early detection of hemodynamic responses using EEG: A hybrid EEG-fNIRS study[J]. Frontiers in Human Neuroscience, 2018, 12: 749. doi: 10.3389/fnhum.2018.00479.
[39]	SUN Zhe, HUANG Zihao, DUAN Feng, et al. A novel multimodal approach for hybrid brain–computer interface[J]. IEEE Access, 2020, 8: 89909–89918. doi: 10.1109/ACCESS.2020.2994226.
[40]	NOUR M, ÖZTURK S, and POLAT K. A novel classification framework using multiple bandwidth method with optimized CNN for brain-computer interfaces with EEG-fNIRS signals[J]. Neural Computing & Applications, 2021, 22(33): 15815–15829. doi: 10.1007/s00521-021-06202-4.
[41]	MUGHAL N E, KHAN M J, KHALIL K, et al. EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM[J]. Frontiers in Neurorobotics, 2022, 16: 873239. doi: 10.3389/FNBOT.2022.873239.
[42]	CHEN Jiaming, WANG Dan, HU Bo, et al. MCFHNet: Multi-channel fusion hybrid network for efficient EEG-fNIRS multi-modal motor imagery decoding[C]. 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, UK, 2022: 4821–4825.
[43]	LI Yang, ZHANG Xin, and MING Dong. Early-stage fusion of EEG and fNIRS improves classification of motor imagery[J]. Frontiers in Neuroscience, 2023, 16: 1062889. doi: 10.3389/fnins.2022.1062889.
[44]	GAO Yunyuan, LIU Hongming, FANG Feng, et al. Classification of working memory loads via assessing broken detailed balance of EEG-FNIRS neurovascular coupling measures[J]. IEEE Transactions on Biomedical Engineering, 2023, 70(3): 877–887. doi: 10.1109/TBME.2022.3204718.
[45]	QIU Lina, ZHOU Yongshi, HE Zhipeng, et al. Improved classification performance of EEG-fNIRS multimodal brain-computer interface based on multi-domain features and multi-level progressive learning[J]. Frontiers in Human Neuroscience, 2022, 16: 973959. doi: 10.3389/fnhum.2022.973959.