A Review of Research Progress on Brain-Computer Interface Systems for Rapid Serial Visual Presentation Based on ElectroEncephaloGram

WEI Wei; QIU Shuang; LI Xujin; MAO Jiayu; WANG Yanzi; HE Huiguang

doi:10.11999/JEIT230952

Volume 46 Issue 2

Feb. 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2024 > 46(2): 443-455

WEI Wei, QIU Shuang, LI Xujin, MAO Jiayu, WANG Yanzi, HE Huiguang. A Review of Research Progress on Brain-Computer Interface Systems for Rapid Serial Visual Presentation Based on ElectroEncephaloGram[J]. Journal of Electronics & Information Technology, 2024, 46(2): 443-455. doi: 10.11999/JEIT230952

Citation:

WEI Wei, QIU Shuang, LI Xujin, MAO Jiayu, WANG Yanzi, HE Huiguang. A Review of Research Progress on Brain-Computer Interface Systems for Rapid Serial Visual Presentation Based on ElectroEncephaloGram[J]. Journal of Electronics & Information Technology, 2024, 46(2): 443-455. doi: 10.11999/JEIT230952

Citation:

WEI Wei, QIU Shuang, LI Xujin, MAO Jiayu, WANG Yanzi, HE Huiguang. A Review of Research Progress on Brain-Computer Interface Systems for Rapid Serial Visual Presentation Based on ElectroEncephaloGram[J]. Journal of Electronics & Information Technology, 2024, 46(2): 443-455. doi: 10.11999/JEIT230952

PDF( 5723 KB)

A Review of Research Progress on Brain-Computer Interface Systems for Rapid Serial Visual Presentation Based on ElectroEncephaloGram

doi: 10.11999/JEIT230952 cstr: 32379.14.JEIT230952

WEI Wei¹,
QIU Shuang^{1, 2},
LI Xujin^{1, 2},
MAO Jiayu^{1, 2},
WANG Yanzi^{1, 2},
HE Huiguang^{1, 2
,
,}

1.
Laboratory of Brain Atlas and Brain-inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100049, China

Funds: The National Natural Science Foundation of China (62206285, U21A20388, 62020106015), General Program of China Postdoctoral Science Foundation (2021M703490)

Received Date: 2023-03-10
Rev Recd Date: 2023-11-30

Available Online: 2023-12-06

Publish Date: 2024-02-29

Abstract

Abstract

Brain-Computer Interface (BCI) system establishes a direct communication pathway between the brain and external devices, and combined with the Rapid Serial Visual Presentation (RSVP) paradigm, it can achieve high-throughput target image retrieval by utilizing the human visual system. In recent years, the RSVP-BCI system has made significant progress in research on paradigm, ElectroEncephaloGram (EEG) decoding, and system applications. Research on paradigm reveals the impact of different paradigm parameters on system performance, promoting the improvement of system performance; The research on EEG decoding improves the classification performance of algorithms and promotes applications in scenarios such as few training, zero training samples, and multimodality; The research on the RSVP-BCI system application has driven the system towards practical applications and expanded its application fields. However, the system also faces challenges such as limited practical applications, difficulties in cross-domain decoding of EEG, and the rapid progress of computer vision. This article reviews and summarizes the research progress of RSVP-BCI in recent years, and looks forward to the future development direction.
- Brain-Computer Interface (BCI),
- Rapid Serial Visual Presentation (RSVP),
- ElectroEncephaloGram (EEG)

FullText(HTML)

References(62)

References

[1]	WOLPAW J R, WOLPAW E W, 伏云发, 杨秋红, 徐保磊, 等译. 脑-机接口: 原理与实践[M]. 北京: 国防工业出版社, 2017: 3. WOLPAW J R, WOLPAW E W, FU Yunfa, YANG Qiuhong, XU Baolei, et al. translation. Brain-Computer Interface: Principles and Practice[M]. Beijing: National Defense Industry Press, 2017: 3.
[2]	GERSON A D, PARRA L C, and SAJDA P. Cortically coupled computer vision for rapid image search[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2006, 14(2): 174–179. doi: 10.1109/TNSRE.2006.875550.
[3]	LEES S, DAYAN N, CE - computer interfaces[J]. Journal of Neural Engineering, 2018, 15(2): 021001. doi: 10.1088/1741-2552/COTTI H, et al. A review of rapid serial visual presentation-based brainaa9817.
[4]	LUCK S J, 范思陆, 丁玉珑, 曲折, 等译. 事件相关电位基础[M]. 上海: 华东师范大学出版社, 2009: 6. LUCK S J, FAN Silu, DING Yulong, QU Zhe, et al. translation. Introduction to the Event-Related Potential Technique[M]. Shanghai: East China Normal University Press, 2009: 6.
[5]	YI Weibo, QIU Shuang, FAN Xinan, et al. Estimation of mental workload induced by different presentation rates in rapid serial visual presentation tasks[C]. 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Beilin, Germany, 2019: 5552–5555. doi: 10.1109/EMBC.2019.8857274.
[6]	CECOTTI H. Single-trial detection with magnetoencephalography during a dual-rapid serial visual presentation task[J]. IEEE Transactions on Biomedical Engineering, 2016, 63(1): 220–227. doi: 10.1109/TBME.2015.2478695.
[7]	LIN Zhimin, ZHANG Chi, ZENG Ying, et al. A novel P300 BCI speller based on the Triple RSVP paradigm[J]. Scientific Reports, 2018, 8(1): 3350. doi: 10.1038/s41598-018-21717-y.
[8]	LIN Zhimin, ZENG Ying, WANG Xiaojuan, et al. EEG-based target detection during a multi-rapid serial visual presentation[C]. 8th International IEEE/EMBS Conference on Neural Engineering (NER), Shanghai, China, 2017: 556–559. doi: 10.1109/NER.2017.8008412.
[9]	ZHANG Shangen, CHEN, Xiaogang, WANG Yijun, et al. Visual field inhomogeneous in brain–computer interfaces based on rapid serial visual presentation[J]. Journal of Neural Engineering, 2022, 19(1): 016015. doi: 10.1088/1741-2552/ac4a3e.
[10]	MATRAN-FERNANDEZ A and POLI R. Brain–computer interfaces for detection and localization of targets in aerial images[J]. IEEE Transactions on Biomedical Engineering, 2017, 64(4): 959–969. doi: 10.1109/TBME.2016.2583200.
[11]	WON D O, HWANG H J, KIM D M, et al. Motion-based rapid serial visual presentation for gaze-independent brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018, 26(2): 334–343. doi: 10.1109/TNSRE.2017.2736600.
[12]	MIJANI A M, SHAMSOLLAHI M B, and HASSANI M S. A novel dual and triple shifted RSVP paradigm for P300 speller[J]. Journal of Neuroscience Methods, 2019, 328: 108420. doi: 10.1016/j.jneumeth.2019.108420.
[13]	JALILPOUR S, SARDOUIE S H, and MIJANI A. A novel hybrid BCI speller based on RSVP and SSVEP paradigm[J]. Computer Methods and Programs in Biomedicine, 2020, 187: 105326. doi: 10.1016/j.cmpb.2020.105326.
[14]	RIVET B, SOULOUMIAC A, ATTINA V, et al. xDAWN algorithm to enhance evoked potentials: Application to brain–computer interface[J]. IEEE Transactions on Biomedical Engineering, 2009, 56(8): 2035–2043. doi: 10.1109/TBME.2009.2012869.
[15]	BARACHANT A and CONGEDO M. A plug&play P300 BCI using information geometry[EB/OL]. https://arxiv.org/abs/1409.0107, 2014.
[16]	MANOR R and GEVA A B. Convolutional neural network for multi-category rapid serial visual presentation BCI[J]. Frontiers in Computational Neuroscience, 2015, 9: 168707. doi: 10.3389/fncom.2015.00146.
[17]	LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGNet: A compact convolutional neural network for EEG-based brain–computer interfaces[J]. Journal of Neural Engineering, 2018, 15(5): 056013. doi: 10.1088/1741-2552/aace8c.
[18]	XIAO Xiaolin, XU Minpeng, JIN Jing, et al. Discriminative canonical pattern matching for single-trial classification of ERP components[J]. IEEE Transactions on Biomedical Engineering, 2020, 67(8): 2266–2275. doi: 10.1109/TBME.2019.2958641.
[19]	CUI Yujie, XIE Songyun, XIE Xinzhou, et al. LDER: A classification framework based on ERP enhancement in RSVP task[J]. Journal of Neural Engineering, 2023, 20(3): 036029. doi: 10.1088/1741-2552/acd95d.
[20]	LI Bowen, ZHANG Shangen, HU Yijun, et al. Assembling global and local spatial-temporal filters to extract discriminant information of EEG in RSVP task[J]. Journal of Neural Engineering, 2023, 20(1): 016052. doi: 10.1088/1741-2552/acb96f.
[21]	LI Bowen, LIN Yanfei, GAO Xiaorong, et al. Enhancing the EEG classification in RSVP task by combining interval model of ERPs with spatial and temporal regions of interest[J]. Journal of Neural Engineering, 2021, 18(1): 016008. doi: 10.1088/1741-2552/abc8d5.
[22]	SHAN Hongchang, LIU Yu, and STEFANOV T. A simple convolutional neural network for accurate P300 detection and character spelling in brain computer interface[C]. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018: 1604–1610.
[23]	SANTAMARIA-VAZQUEZ E, MARTINEZ-CAGIGAL V, VAQUERIZO-VILLAR F, et al. EEG-inception: A novel deep convolutional neural network for assistive ERP-based brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(12): 2773–2782. doi: 10.1109/TNSRE.2020.3048106.
[24]	MA Ronghua, YU Tianyou, ZHONG Xiaoli, et al. Capsule network for ERP detection in brain-computer interface[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 718–730. doi: 10.1109/TNSRE.2021.3070327.
[25]	MAO Jiayu, QIU Shuang, WEI Wei, et al. Cross-modal guiding and reweighting network for multi-modal RSVP-based target detection[J]. Neural Networks, 2023, 161: 65–82. doi: 10.1016/j.neunet.2023.01.009.
[26]	ZANG Boyu, LIN Yanfei, LIU Zhiwei, et al. A deep learning method for single-trial EEG classification in RSVP task based on spatiotemporal features of ERPs[J]. Journal of Neural Engineering, 2021, 18(4): 0460c8. doi: 10.1088/1741-2552/ac1610.
[27]	LI Fu, WANG Chong, LI Yang, et al. Phase preservation neural network for electroencephalography classification in rapid serial visual presentation task[J]. IEEE Transactions on Biomedical Engineering, 2022, 69(6): 1931–1942. doi: 10.1109/TBME.2021.3130917.
[28]	JAYARAM V, ALAMGIR M, ALTUN Y, et al. Transfer learning in brain-computer interfaces[J]. IEEE Computa tional Intelligence Magazine, 2016, 11(1): 20–31. doi: 10.1109/MCI.2015.2501545.
[29]	SHAMWELL J, LEE H, KWON H, et al. Single-trial EEG RSVP classification using convolutional neural networks[C]. Proceedings of the SPIE 9836, Micro-and Nanotechnology Sensors, Systems, and Applications VIII, Baltimore, USA, 2016: 983622. doi: 10.1117/12.2224172.
[30]	HAJINOROOZI M, MAO Zijin, LIN Yuanpin, et al. Deep transfer learning for cross-subject and cross-experiment prediction of image rapid serial visual presentation events from EEG data[C]. 11th International Conference on Augmented Cognition, Vancouver, Canada, 2017: 45–55. doi: 10.1007/978-3-319-58628-1_4.
[31]	MIJANI A M, EINIZADE A, SHAMSOLLAHI M B, et al. Cross-subject and cross-paradigm learning using convolutional neural network for P300 event-related potential detection[J]. Journal of Neurology and Neuroscience, 2020, 11(5): 329. doi: 10.36648/2171-6625.11.1.329.
[32]	ZHANG Wen and WU Dongrui. Manifold embedded knowledge transfer for brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(5): 1117–1127. doi: 10.1109/TNSRE.2020.2985996.
[33]	WEI Wei, QIU Shuang, MA Xuelin, et al. Reducing calibration efforts in RSVP tasks with multi-source adversarial domain adaptation[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(11): 2344–2355. doi: 10.1109/TNSRE.2020.3023761.
[34]	FAN Liangwei, SHEN Hui, XIE Fengyu, et al. DC-tCNN: A deep model for EEG-based detection of dim targets[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 1727–1736. doi: 10.1109/TNSRE.2022.3184725.
[35]	SHE Qingshan, CAI Yinhao, DU Shengzhi, et al. Multi-source manifold feature transfer learning with domain selection for brain-computer interfaces[J]. Neurocomputing, 2022, 514: 313–327. doi: 10.1016/j.neucom.2022.09.124.
[36]	WAYTOWICH N R, LAWHERN V J, BOHANNON A W, et al. Spectral transfer learning using information geometry for a user-independent brain-computer interface[J]. Frontiers in Neuroscience, 2016, 10: 430. doi: 10.3389/fnins.2016.00430.
[37]	LEE J, WON K, KWON M, et al. CNN with large data achieves true zero-training in online P300 brain-computer interface[J]. IEEE Access, 2020, 8: 74385–74400. doi: 10.1109/ACCESS.2020.2988057.
[38]	WEI Wei, QIU Shuang, ZHANG Yukun, et al. ERP prototypical matching net: A meta-learning method for zero-calibration RSVP-based image retrieval[J]. Journal of Neural Engineering, 2022, 19(2): 026028. doi: 10.1088/1741-2552/ac5eb7.
[39]	LI Xujin, QIU Shuang, WEI Wei, et al. A zero-training method for RSVP-based brain computer interface[C]. Chinese Conference on Pattern Recognition and Computer Vision (PRCV), Shenzhen, China, 2022: 113–125. doi: 10.1007/978-3-031-18910-4_10.
[40]	LI Xujin, WEI Wei, QIU Shuang, et al. TFF-Former: Temporal-frequency fusion transformer for zero-training decoding of two BCI tasks[C]. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal, 2022: 51–59. doi: 10.1145/3503161.3548269.
[41]	MANOR R, MISHALI L, and GEVA A B. Multimodal neural network for rapid serial visual presentation brain computer interface[J]. Frontiers in Computational Neuroscience, 2016, 10: 130. doi: 10.3389/fncom.2016.00130.
[42]	WU Qunjian, ZENG Ying, ZHANG Chi, et al. An EEG-based person authentication system with open-set capability combining eye blinking signals[J]. Sensors, 2018, 18(2): 335. doi: 10.3390/s18020335.
[43]	DING Yi, HUYNH B, XU Aiwen, et al. Multimodal classification of EEG during physical activity[C]. International Conference on Multimodal Interaction, Suzhou, China, 2019, 185–194. doi: 10.1145/3340555.3353759.
[44]	MATRAN-FERNANDEZ A and POLI R. Towards the automated localisation of targets in rapid image-sifting by collaborative brain-computer interfaces[J]. PLoS One, 2017, 12(5): e0178498. doi: 10.1371/journal.pone.0178498.
[45]	ZHENG Li, SUN Sen, ZHAO Hongze, et al. A cross-session dataset for collaborative brain-computer interfaces based on rapid serial visual presentation[J]. Front Neuroscience, 2020, 14: 579469. doi: 10.3389/fnins.2020.579469.
[46]	ZHANG Hangkui, ZHU Li, XU Senwei, et al. Two brains, one target: Design of a multi-level information fusion model based on dual-subject RSVP[J]. Journal of Neuroscience Methods, 2021, 363: 109346. doi: 10.1016/j.jneumeth.2021.109346.
[47]	ZHAO Ziwei, LIN Yanfei, WANG Yijun, et al. Single-trial EEG classification using spatio-temporal weighting and correlation analysis for RSVP-based collaborative brain computer interface[J]. Transactions on Biomedical Engineering, 2023. doi: 10.1109/TBME.2023.3309255.
[48]	XU Meng, CHEN Yuanfang, WANG Dan, et al. Multi-objective optimization approach for channel selection and cross-subject generalization in RSVP-based BCIs[J]. Journal of Neural Engineering, 2021, 18(4): 046076. doi: 10.1088/1741-2552/ac0489.
[49]	HE Chao, LIU Jialu, ZHU Yuesheng, et al. Data augmentation for deep neural networks model in EEG classification task: A review[J]. Frontiers in Human Neuroscience, 2021, 15: 765525. doi: 10.3389/fnhum.2021.765525.
[50]	PANWAR S, RAD P, QUARLES J, et al. Generating EEG signals of an RSVP experiment by a class conditioned Wasserstein generative adversarial network[C]. 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Bari, Italy, 2019: 1304–1310. doi: 10.1109/SMC.2019.8914492.
[51]	PANWAR S, RAD P, JUNG T P, et al. Modeling EEG data distribution with a Wasserstein generative adversarial network to predict RSVP events[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(8): 1720–1730. doi: 10.1109/TNSRE.2020.3006180.
[52]	XU Meng, CHEN Yuanfang, WANG Yijun, et al. BWGAN-GP: An EEG data generation method for class imbalance problem in RSVP tasks[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 251–263. doi: 10.1109/TNSRE.2022.3145515.
[53]	CUI Yujie, XIE Songyun, XIE Xinzhou, et al. Dynamic probability integration for electroencephalography-based rapid serial visual presentation performance enhancement: Application in nighttime vehicle detection[J]. Frontiers in Computational Neuroscience, 2022, 16: 1006361. doi: 10.3389/fncom.2022.1006361.
[54]	WU Qunjian, YAN Bin, ZENG Ying, et al. Anti-deception: Reliable EEG-based biometrics with real-time capability from the neural response of face rapid serial visual presentation[J]. BioMedical Engineering OnLine, 2018, 17(1): 55. doi: 10.1186/s12938-018-0483-7.
[55]	ZENG Ying, WU Qunjian, YANG Kai, et al. EEG-based identity authentication framework using face rapid serial visual presentation with optimized channels[J]. Sensors (Basel), 2018, 19(1): 6. doi: 10.3390/s19010006.
[56]	WANG Hanwen, QI Yu, YU Hang, et al. , RCIT: An RSVP-based concealed information test framework using EEG signals[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022, 14(2): 541–551. doi: 10.1109/TCDS.2021.3053455.
[57]	NAYAK T, KO L W, JUNG T P, et al. Target classification in a novel SSVEP-RSVP based BCI gaming system[C]. 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Bari, Italy, 2019: 4194–4198. doi: 10.1109/SMC.2019.8914174.
[58]	KO L W, SANKAR D S V, HUANG Yufei, et al. SSVEP-assisted RSVP brain–computer interface paradigm for multi-target classification[J]. Journal of Neural Engineering, 2021, 18(1): 016021. doi: 10.1088/1741-2552/abd1c0.
[59]	ACKERMAN E and STRICKLAND E. Are you ready for workplace brain scanning?[EB/OL]. https://spectrum.ieee.org/neurotech-workplace-innereye-emotiv, 2022.
[60]	YI Weibo, QIU Shuang, FAN Xinan, et al. Evaluation of mental workload associated with time pressure in rapid serial visual presentation tasks[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022, 14(2): 608–616. doi: 10.1109/tcds.2021.3061564.
[61]	TANG M F, FORD L, ARABZADEH E, et al. Neural dynamics of the attentional blink revealed by encoding orientation selectivity during rapid visual presentation[J]. Nature Communications, 2020, 11(1): 434. doi: 10.1038/s41467-019-14107-z.
[62]	SUN Meng, LIU Fang, CUI Lixia, et al. The effect of fearful faces on the attentional blink is modulated by emotional task relevance: An event-related potential study[J]. Neuropsychologia, 2021, 162: 108043. doi: 10.1016/j.neuropsychologia.2021.108043.