高级搜索

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于脑电的快速序列视觉呈现脑-机接口系统研究进展综述

魏玮 邱爽 李叙锦 毛嘉宇 王妍紫 何晖光

魏玮, 邱爽, 李叙锦, 毛嘉宇, 王妍紫, 何晖光. 基于脑电的快速序列视觉呈现脑-机接口系统研究进展综述[J]. 电子与信息学报, 2024, 46(2): 443-455. doi: 10.11999/JEIT230952
引用本文: 魏玮, 邱爽, 李叙锦, 毛嘉宇, 王妍紫, 何晖光. 基于脑电的快速序列视觉呈现脑-机接口系统研究进展综述[J]. 电子与信息学报, 2024, 46(2): 443-455. doi: 10.11999/JEIT230952
WEI Wei, QIU Shuang, LI Xujin, MAO Jiayu, WANG Yanzi, HE Huiguang. A Review of Research Progress on Brain-Computer Interface Systems for Rapid Serial Visual Presentation Based on ElectroEncephaloGram[J]. Journal of Electronics & Information Technology, 2024, 46(2): 443-455. doi: 10.11999/JEIT230952
Citation: WEI Wei, QIU Shuang, LI Xujin, MAO Jiayu, WANG Yanzi, HE Huiguang. A Review of Research Progress on Brain-Computer Interface Systems for Rapid Serial Visual Presentation Based on ElectroEncephaloGram[J]. Journal of Electronics & Information Technology, 2024, 46(2): 443-455. doi: 10.11999/JEIT230952

基于脑电的快速序列视觉呈现脑-机接口系统研究进展综述

doi: 10.11999/JEIT230952
基金项目: 国家自然科学基金(62206285, U21A20388, 62020106015),中国博士后科学基金(2021M703490)
详细信息
    作者简介:

    魏玮:男,助理研究员,研究方向为脑-机接口、模式识别

    邱爽:女,副研究员,研究方向为脑-机接口、人机交互

    李叙锦:男,博士生,研究方向为脑-机接口、深度学习

    毛嘉宇:男,博士生,研究方向为脑-机接口、多模态机器学习

    王妍紫:女,硕士生,研究方向为脑-机接口、认知负荷

    何晖光:男,研究员,研究方向为脑-机接口、模式识别与人工智能、医学影像处理

    通讯作者:

    何晖光 huiguang.he@ia.ac.cn

  • 中图分类号: TN911.7; TP391

A Review of Research Progress on Brain-Computer Interface Systems for Rapid Serial Visual Presentation Based on ElectroEncephaloGram

Funds: The National Natural Science Foundation of China (62206285, U21A20388, 62020106015), General Program of China Postdoctoral Science Foundation (2021M703490)
  • 摘要: 脑-机接口(BCI)系统建立大脑与外部设备之间的直接交流通路,结合快速序列视觉呈现(RSVP)范式能够实现利用人类视觉系统进行高流通量图像目标检索。近些年来,RSVP-BCI系统在范式编码、脑电(EEG)解码和系统应用方面的研究取得了长足的进步。对范式编码的研究揭示不同范式参数对系统性能的影响,促进提升系统性能;脑电解码的研究在提升算法分类性能的同时推动少训练、零训练样本、多模态等场景下的应用;对RSVP-BCI系统应用的研究实现推动系统走向实际应用并拓宽了应用领域。同时,系统仍面临着迈向实际时可应用领域范围窄、脑电跨域解码难题以及计算机视觉飞速进步带来的挑战。该文对RSVP-BCI近年来的相关研究进展进行了回顾与总结,并对未来的发展方向进行了展望。
  • 图  1  RSVP-BCI系统示意图

    图  2  基于RSVP的目标检索任务中多名被试平均的ERP波形

    图  3  双视觉通路RSVP范式示意图(引自文献 [6])

    图  4  实现从源域到目标域的知识迁移存在两种方法[28]

    图  5  RSVP-BCI系统部分应用示意图

    表  1  RSVP编码研究进展简表

    文献 时间 范式因素 应用 研究内容和主要结论
    [5] 2019年 呈现速率 目标检索 研究呈现速率与认知负荷之间的关系,结果显示随着呈现速率的增加,人员的认知负荷增加,任务性能降低。
    [6] 2015年 呈现模式 目标检索 研究了一种左右排列的双视觉通路RSVP,实现优于单视觉通路RSVP的性能。
    [7,8] 2018/2017年 呈现模式 目标检索 研究了单/双三视觉通路RSVP,分别实现0.926,0.946和0.952的分类AUC。
    [9] 2022年 呈现模式 目标检索 研究了双/三视觉通路范式下视野对RSVP目标检索的影响,研究结果表明视野对目标检索性能有显著影响,中心视野优于周边视野,左视野高于右视野,上视野优于下视野。
    [11] 2017年 呈现模式 拼写器 研究了字符随机方向运动的RSVP字符拼写器范式,实现诱发更强的P300信号和拼写器字符识别准确率提升。
    [12] 2019年 呈现模式 拼写器 研究了双/三视觉通路RSVP拼写器,实现提高系统ITR,其中三视觉通路最高实现了在线平均ITR为20.26 bpm。
    [13] 2020年 混合范式 拼写器 研究了RSVP拼写器中的第1个混合范式——RSVP-SSVEP BCI,实现平均信息传输率达到23.41 bpm。
    下载: 导出CSV

    表  2  零校准脑电解码性能(%)

    文献 方法名称 时间 均衡精度
    [36] STIG 2016年 65.82 ± 7.76
    [37] 2020年 85.03 ± 4.70
    [38] EPMN 2022年 86.34 ± 3.54
    [39] 2022年 86.76 ± 3.93
    [40] TFF-Former 2022年 88.05 ± 3.73
    下载: 导出CSV

    表  3  多模态脑电解码性能对比

    文献 时间 任务类型 模态 单模态性能 多模态性能
    [41] 2016年 图像目标检索 1. 脑电
    2. 图像
    [脑电] BA: ~74%
    [图像] BA: ~81%
    BA:85.06%
    [25] 2023年 图像目标检索 1. 脑电
    2. 眼动
    [脑电] BA: 81.85%
    [眼动] BA: 66.97%
    BA: 88.00%
    [42] 2018年 身份认证 1. 脑电
    2. 眼电
    [脑电] Accuracy: 92.40% Accuracy: 97.60%
    [43] 2019年 目标检索 1. 脑电
    2. 眼电
    3. 肌电
    [脑电] F1-score: 0.602 F1-score: 0.657
    下载: 导出CSV
  • [1] WOLPAW J R, WOLPAW E W, 伏云发, 杨秋红, 徐保磊, 等译. 脑-机接口: 原理与实践[M]. 北京: 国防工业出版社, 2017: 3.

    WOLPAW J R, WOLPAW E W, FU Yunfa, YANG Qiuhong, XU Baolei, et al. translation. Brain-Computer Interface: Principles and Practice[M]. Beijing: National Defense Industry Press, 2017: 3.
    [2] GERSON A D, PARRA L C, and SAJDA P. Cortically coupled computer vision for rapid image search[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2006, 14(2): 174–179. doi: 10.1109/TNSRE.2006.875550.
    [3] LEES S, DAYAN N, CE - computer interfaces[J]. Journal of Neural Engineering, 2018, 15(2): 021001. doi: 10.1088/1741-2552/COTTI H, et al. A review of rapid serial visual presentation-based brainaa9817.
    [4] LUCK S J, 范思陆, 丁玉珑, 曲折, 等译. 事件相关电位基础[M]. 上海: 华东师范大学出版社, 2009: 6.

    LUCK S J, FAN Silu, DING Yulong, QU Zhe, et al. translation. Introduction to the Event-Related Potential Technique[M]. Shanghai: East China Normal University Press, 2009: 6.
    [5] YI Weibo, QIU Shuang, FAN Xinan, et al. Estimation of mental workload induced by different presentation rates in rapid serial visual presentation tasks[C]. 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Beilin, Germany, 2019: 5552–5555. doi: 10.1109/EMBC.2019.8857274.
    [6] CECOTTI H. Single-trial detection with magnetoencephalography during a dual-rapid serial visual presentation task[J]. IEEE Transactions on Biomedical Engineering, 2016, 63(1): 220–227. doi: 10.1109/TBME.2015.2478695.
    [7] LIN Zhimin, ZHANG Chi, ZENG Ying, et al. A novel P300 BCI speller based on the Triple RSVP paradigm[J]. Scientific Reports, 2018, 8(1): 3350. doi: 10.1038/s41598-018-21717-y.
    [8] LIN Zhimin, ZENG Ying, WANG Xiaojuan, et al. EEG-based target detection during a multi-rapid serial visual presentation[C]. 8th International IEEE/EMBS Conference on Neural Engineering (NER), Shanghai, China, 2017: 556–559. doi: 10.1109/NER.2017.8008412.
    [9] ZHANG Shangen, CHEN, Xiaogang, WANG Yijun, et al. Visual field inhomogeneous in brain–computer interfaces based on rapid serial visual presentation[J]. Journal of Neural Engineering, 2022, 19(1): 016015. doi: 10.1088/1741-2552/ac4a3e.
    [10] MATRAN-FERNANDEZ A and POLI R. Brain–computer interfaces for detection and localization of targets in aerial images[J]. IEEE Transactions on Biomedical Engineering, 2017, 64(4): 959–969. doi: 10.1109/TBME.2016.2583200.
    [11] WON D O, HWANG H J, KIM D M, et al. Motion-based rapid serial visual presentation for gaze-independent brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018, 26(2): 334–343. doi: 10.1109/TNSRE.2017.2736600.
    [12] MIJANI A M, SHAMSOLLAHI M B, and HASSANI M S. A novel dual and triple shifted RSVP paradigm for P300 speller[J]. Journal of Neuroscience Methods, 2019, 328: 108420. doi: 10.1016/j.jneumeth.2019.108420.
    [13] JALILPOUR S, SARDOUIE S H, and MIJANI A. A novel hybrid BCI speller based on RSVP and SSVEP paradigm[J]. Computer Methods and Programs in Biomedicine, 2020, 187: 105326. doi: 10.1016/j.cmpb.2020.105326.
    [14] RIVET B, SOULOUMIAC A, ATTINA V, et al. xDAWN algorithm to enhance evoked potentials: Application to brain–computer interface[J]. IEEE Transactions on Biomedical Engineering, 2009, 56(8): 2035–2043. doi: 10.1109/TBME.2009.2012869.
    [15] BARACHANT A and CONGEDO M. A plug&play P300 BCI using information geometry[EB/OL]. https://arxiv.org/abs/1409.0107, 2014.
    [16] MANOR R and GEVA A B. Convolutional neural network for multi-category rapid serial visual presentation BCI[J]. Frontiers in Computational Neuroscience, 2015, 9: 168707. doi: 10.3389/fncom.2015.00146.
    [17] LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGNet: A compact convolutional neural network for EEG-based brain–computer interfaces[J]. Journal of Neural Engineering, 2018, 15(5): 056013. doi: 10.1088/1741-2552/aace8c.
    [18] XIAO Xiaolin, XU Minpeng, JIN Jing, et al. Discriminative canonical pattern matching for single-trial classification of ERP components[J]. IEEE Transactions on Biomedical Engineering, 2020, 67(8): 2266–2275. doi: 10.1109/TBME.2019.2958641.
    [19] CUI Yujie, XIE Songyun, XIE Xinzhou, et al. LDER: A classification framework based on ERP enhancement in RSVP task[J]. Journal of Neural Engineering, 2023, 20(3): 036029. doi: 10.1088/1741-2552/acd95d.
    [20] LI Bowen, ZHANG Shangen, HU Yijun, et al. Assembling global and local spatial-temporal filters to extract discriminant information of EEG in RSVP task[J]. Journal of Neural Engineering, 2023, 20(1): 016052. doi: 10.1088/1741-2552/acb96f.
    [21] LI Bowen, LIN Yanfei, GAO Xiaorong, et al. Enhancing the EEG classification in RSVP task by combining interval model of ERPs with spatial and temporal regions of interest[J]. Journal of Neural Engineering, 2021, 18(1): 016008. doi: 10.1088/1741-2552/abc8d5.
    [22] SHAN Hongchang, LIU Yu, and STEFANOV T. A simple convolutional neural network for accurate P300 detection and character spelling in brain computer interface[C]. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018: 1604–1610.
    [23] SANTAMARIA-VAZQUEZ E, MARTINEZ-CAGIGAL V, VAQUERIZO-VILLAR F, et al. EEG-inception: A novel deep convolutional neural network for assistive ERP-based brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(12): 2773–2782. doi: 10.1109/TNSRE.2020.3048106.
    [24] MA Ronghua, YU Tianyou, ZHONG Xiaoli, et al. Capsule network for ERP detection in brain-computer interface[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 718–730. doi: 10.1109/TNSRE.2021.3070327.
    [25] MAO Jiayu, QIU Shuang, WEI Wei, et al. Cross-modal guiding and reweighting network for multi-modal RSVP-based target detection[J]. Neural Networks, 2023, 161: 65–82. doi: 10.1016/j.neunet.2023.01.009.
    [26] ZANG Boyu, LIN Yanfei, LIU Zhiwei, et al. A deep learning method for single-trial EEG classification in RSVP task based on spatiotemporal features of ERPs[J]. Journal of Neural Engineering, 2021, 18(4): 0460c8. doi: 10.1088/1741-2552/ac1610.
    [27] LI Fu, WANG Chong, LI Yang, et al. Phase preservation neural network for electroencephalography classification in rapid serial visual presentation task[J]. IEEE Transactions on Biomedical Engineering, 2022, 69(6): 1931–1942. doi: 10.1109/TBME.2021.3130917.
    [28] JAYARAM V, ALAMGIR M, ALTUN Y, et al. Transfer learning in brain-computer interfaces[J]. IEEE Computa tional Intelligence Magazine, 2016, 11(1): 20–31. doi: 10.1109/MCI.2015.2501545.
    [29] SHAMWELL J, LEE H, KWON H, et al. Single-trial EEG RSVP classification using convolutional neural networks[C]. Proceedings of the SPIE 9836, Micro-and Nanotechnology Sensors, Systems, and Applications VIII, Baltimore, USA, 2016: 983622. doi: 10.1117/12.2224172.
    [30] HAJINOROOZI M, MAO Zijin, LIN Yuanpin, et al. Deep transfer learning for cross-subject and cross-experiment prediction of image rapid serial visual presentation events from EEG data[C]. 11th International Conference on Augmented Cognition, Vancouver, Canada, 2017: 45–55. doi: 10.1007/978-3-319-58628-1_4.
    [31] MIJANI A M, EINIZADE A, SHAMSOLLAHI M B, et al. Cross-subject and cross-paradigm learning using convolutional neural network for P300 event-related potential detection[J]. Journal of Neurology and Neuroscience, 2020, 11(5): 329. doi: 10.36648/2171-6625.11.1.329.
    [32] ZHANG Wen and WU Dongrui. Manifold embedded knowledge transfer for brain-computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(5): 1117–1127. doi: 10.1109/TNSRE.2020.2985996.
    [33] WEI Wei, QIU Shuang, MA Xuelin, et al. Reducing calibration efforts in RSVP tasks with multi-source adversarial domain adaptation[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(11): 2344–2355. doi: 10.1109/TNSRE.2020.3023761.
    [34] FAN Liangwei, SHEN Hui, XIE Fengyu, et al. DC-tCNN: A deep model for EEG-based detection of dim targets[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 1727–1736. doi: 10.1109/TNSRE.2022.3184725.
    [35] SHE Qingshan, CAI Yinhao, DU Shengzhi, et al. Multi-source manifold feature transfer learning with domain selection for brain-computer interfaces[J]. Neurocomputing, 2022, 514: 313–327. doi: 10.1016/j.neucom.2022.09.124.
    [36] WAYTOWICH N R, LAWHERN V J, BOHANNON A W, et al. Spectral transfer learning using information geometry for a user-independent brain-computer interface[J]. Frontiers in Neuroscience, 2016, 10: 430. doi: 10.3389/fnins.2016.00430.
    [37] LEE J, WON K, KWON M, et al. CNN with large data achieves true zero-training in online P300 brain-computer interface[J]. IEEE Access, 2020, 8: 74385–74400. doi: 10.1109/ACCESS.2020.2988057.
    [38] WEI Wei, QIU Shuang, ZHANG Yukun, et al. ERP prototypical matching net: A meta-learning method for zero-calibration RSVP-based image retrieval[J]. Journal of Neural Engineering, 2022, 19(2): 026028. doi: 10.1088/1741-2552/ac5eb7.
    [39] LI Xujin, QIU Shuang, WEI Wei, et al. A zero-training method for RSVP-based brain computer interface[C]. Chinese Conference on Pattern Recognition and Computer Vision (PRCV), Shenzhen, China, 2022: 113–125. doi: 10.1007/978-3-031-18910-4_10.
    [40] LI Xujin, WEI Wei, QIU Shuang, et al. TFF-Former: Temporal-frequency fusion transformer for zero-training decoding of two BCI tasks[C]. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal, 2022: 51–59. doi: 10.1145/3503161.3548269.
    [41] MANOR R, MISHALI L, and GEVA A B. Multimodal neural network for rapid serial visual presentation brain computer interface[J]. Frontiers in Computational Neuroscience, 2016, 10: 130. doi: 10.3389/fncom.2016.00130.
    [42] WU Qunjian, ZENG Ying, ZHANG Chi, et al. An EEG-based person authentication system with open-set capability combining eye blinking signals[J]. Sensors, 2018, 18(2): 335. doi: 10.3390/s18020335.
    [43] DING Yi, HUYNH B, XU Aiwen, et al. Multimodal classification of EEG during physical activity[C]. International Conference on Multimodal Interaction, Suzhou, China, 2019, 185–194. doi: 10.1145/3340555.3353759.
    [44] MATRAN-FERNANDEZ A and POLI R. Towards the automated localisation of targets in rapid image-sifting by collaborative brain-computer interfaces[J]. PLoS One, 2017, 12(5): e0178498. doi: 10.1371/journal.pone.0178498.
    [45] ZHENG Li, SUN Sen, ZHAO Hongze, et al. A cross-session dataset for collaborative brain-computer interfaces based on rapid serial visual presentation[J]. Front Neuroscience, 2020, 14: 579469. doi: 10.3389/fnins.2020.579469.
    [46] ZHANG Hangkui, ZHU Li, XU Senwei, et al. Two brains, one target: Design of a multi-level information fusion model based on dual-subject RSVP[J]. Journal of Neuroscience Methods, 2021, 363: 109346. doi: 10.1016/j.jneumeth.2021.109346.
    [47] ZHAO Ziwei, LIN Yanfei, WANG Yijun, et al. Single-trial EEG classification using spatio-temporal weighting and correlation analysis for RSVP-based collaborative brain computer interface[J]. Transactions on Biomedical Engineering, 2023. doi: 10.1109/TBME.2023.3309255.
    [48] XU Meng, CHEN Yuanfang, WANG Dan, et al. Multi-objective optimization approach for channel selection and cross-subject generalization in RSVP-based BCIs[J]. Journal of Neural Engineering, 2021, 18(4): 046076. doi: 10.1088/1741-2552/ac0489.
    [49] HE Chao, LIU Jialu, ZHU Yuesheng, et al. Data augmentation for deep neural networks model in EEG classification task: A review[J]. Frontiers in Human Neuroscience, 2021, 15: 765525. doi: 10.3389/fnhum.2021.765525.
    [50] PANWAR S, RAD P, QUARLES J, et al. Generating EEG signals of an RSVP experiment by a class conditioned Wasserstein generative adversarial network[C]. 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Bari, Italy, 2019: 1304–1310. doi: 10.1109/SMC.2019.8914492.
    [51] PANWAR S, RAD P, JUNG T P, et al. Modeling EEG data distribution with a Wasserstein generative adversarial network to predict RSVP events[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(8): 1720–1730. doi: 10.1109/TNSRE.2020.3006180.
    [52] XU Meng, CHEN Yuanfang, WANG Yijun, et al. BWGAN-GP: An EEG data generation method for class imbalance problem in RSVP tasks[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 251–263. doi: 10.1109/TNSRE.2022.3145515.
    [53] CUI Yujie, XIE Songyun, XIE Xinzhou, et al. Dynamic probability integration for electroencephalography-based rapid serial visual presentation performance enhancement: Application in nighttime vehicle detection[J]. Frontiers in Computational Neuroscience, 2022, 16: 1006361. doi: 10.3389/fncom.2022.1006361.
    [54] WU Qunjian, YAN Bin, ZENG Ying, et al. Anti-deception: Reliable EEG-based biometrics with real-time capability from the neural response of face rapid serial visual presentation[J]. BioMedical Engineering OnLine, 2018, 17(1): 55. doi: 10.1186/s12938-018-0483-7.
    [55] ZENG Ying, WU Qunjian, YANG Kai, et al. EEG-based identity authentication framework using face rapid serial visual presentation with optimized channels[J]. Sensors (Basel), 2018, 19(1): 6. doi: 10.3390/s19010006.
    [56] WANG Hanwen, QI Yu, YU Hang, et al. , RCIT: An RSVP-based concealed information test framework using EEG signals[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022, 14(2): 541–551. doi: 10.1109/TCDS.2021.3053455.
    [57] NAYAK T, KO L W, JUNG T P, et al. Target classification in a novel SSVEP-RSVP based BCI gaming system[C]. 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Bari, Italy, 2019: 4194–4198. doi: 10.1109/SMC.2019.8914174.
    [58] KO L W, SANKAR D S V, HUANG Yufei, et al. SSVEP-assisted RSVP brain–computer interface paradigm for multi-target classification[J]. Journal of Neural Engineering, 2021, 18(1): 016021. doi: 10.1088/1741-2552/abd1c0.
    [59] ACKERMAN E and STRICKLAND E. Are you ready for workplace brain scanning?[EB/OL]. https://spectrum.ieee.org/neurotech-workplace-innereye-emotiv, 2022.
    [60] YI Weibo, QIU Shuang, FAN Xinan, et al. Evaluation of mental workload associated with time pressure in rapid serial visual presentation tasks[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022, 14(2): 608–616. doi: 10.1109/tcds.2021.3061564.
    [61] TANG M F, FORD L, ARABZADEH E, et al. Neural dynamics of the attentional blink revealed by encoding orientation selectivity during rapid visual presentation[J]. Nature Communications, 2020, 11(1): 434. doi: 10.1038/s41467-019-14107-z.
    [62] SUN Meng, LIU Fang, CUI Lixia, et al. The effect of fearful faces on the attentional blink is modulated by emotional task relevance: An event-related potential study[J]. Neuropsychologia, 2021, 162: 108043. doi: 10.1016/j.neuropsychologia.2021.108043.
  • 加载中
图(5) / 表(3)
计量
  • 文章访问数:  1034
  • HTML全文浏览量:  327
  • PDF下载量:  236
  • 被引次数: 0
出版历程
  • 收稿日期:  2023-03-10
  • 修回日期:  2023-11-30
  • 网络出版日期:  2023-12-06
  • 刊出日期:  2024-02-29

目录

    /

    返回文章
    返回