基于卷积神经网络和领域泛化的跨操作员认知负荷识别

周月莹; 公沛良; 王澎湃; 温旭云; 张道强

doi:10.11999/JEIT221491

基于卷积神经网络和领域泛化的跨操作员认知负荷识别

doi: 10.11999/JEIT221491

1.
南京航空航天大学计算机科学与技术学院南京 211106
2.
南京航空航天大学模式分析与机器智能工信部重点实验室南京 211106
3.
南京航空航天大学脑机智能技术教育部重点实验室南京 211106

基金项目: 国家自然科学基金(62136004, 61876082, 61732006)，国家重点研发计划(2018YFC2001600, 2018YFC2001602)，中央高校基本科研业务费专项资金 (NP2022451)

详细信息

作者简介:
周月莹：女，博士生，研究方向为模式识别与脑机接口

公沛良：男，博士生，研究方向为模式识别与脑机接口

王澎湃：男，博士生，研究方向为模式识别与脑机接口

温旭云：女，博士，硕士生导师，研究方向为模式识别、脑机接口

张道强：男，教授，博士生导师，研究方向为模式识别、脑机接口、医疗影像分析

通讯作者:
张道强　dqzhang@nuaa.edu.cn

中图分类号: TN911.7; TP391
计量
- 文章访问数: 810
- HTML全文浏览量: 381
- PDF下载量: 128
- 被引次数: 0
出版历程
- 收稿日期: 2022-11-30
- 修回日期: 2023-05-23
- 网络出版日期: 2023-05-30
- 刊出日期: 2023-08-21

Cross-operator Cognitive Workload Recognition Based on Convolutional Neural Network and Domain Generalization

1.
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
2.
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
3.
Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China

Funds: The National Natural Science Foundation of China (62136004, 61876082, 61732006), The National Key R&D Program of China (2018YFC2001600, 2018YFC2001602), The Fundamental Research Funds for the Central Universities (NP2022451)

摘要

摘要: 基于脑电信号(EEG)的操作员认知负荷识别(CWR)在人机交互系统和被动式脑机接口中有重要价值，然而EEG的非稳态性和被试差异性极大阻碍了跨操作员CWR这一现实场景的快速应用。该文针对跨操作员CWR精度低等问题，提出一种基于卷积神经网络(CNN)和领域泛化(DG)的联合共享特征优化方法(CNN_DG)。该方法通过使用已有操作员(源域)的数据提高未知操作员(目标域)的CWR性能，其主要包括3个模块：深度特征提取器、标签分类器和领域泛化器。深度特征提取器学习可迁移的源域之间的共享知识表征；标签分类器进一步学习深层表征并预测负荷级别；领域泛化器通过与特征提取器进行对抗训练来减少源域间的数据分布差异，从而保证学习特征的共享性。该文在多属性任务组(MATB II)模拟飞行任务竞赛数据集1和2上进行两个三分类的跨操作员CWR实验，并采用留一被试交叉验证策略验证模型识别性能。实验结果表明所提CNN_DG方法显著优于比较方法，验证了其在跨操作员CWR领域的有效性和泛化性。
- 人机交互 /
- 认知负荷 /
- 跨操作员 /
- 卷积神经网络 /
- 领域泛化
Abstract: ElectroEncephaloGraphy (EEG)-based Cognitive Workload Recognition (CWR) is valuable for human-robot interaction systems and passive brain-computer interfaces. However, the none-stationary of EEG and the difference between subjects hinder the rapid application of cross-operator CWR, a realistic scenario. To deal with the above problem, a jointly shared feature optimization method based on the Convolutional Neural Network (CNN) and Domain Generalization (DG) is proposed, denoted as CNN_DG. The data of existing operators (source domains) is used to improve the CWR performance of unknown operators (target domain). It includes three modules: EEG feature extractor, label classifier, and domain generalizer. The EEG feature extractor learns the transferable shared knowledge representation between source domains. The label classifier learns further the deep representation and predicted the workload levels. By adversarial training with the feature extractor, the domain generalizer reduces the difference in source domain distribution and ensures further the sharing of learned features. Two three-categories cross-operator CWR experiments are conducted on the Multi-attribute Task Battery (MATB II) simulated flight competition datasets 1 and 2, and the model performance is verified by using leave-one-subject-out cross-validation. Experimental results showed the CNN_DG performed significantly better than comparing methods, indicating its effectiveness and generalization in the field of cross-operator CWR.
- Human-robot interaction /
- Cognitive workload /
- Cross-operator /
- Convolutional Neural Network (CNN) /
- Domain Generalization (DG)

HTML全文

图 1 跨操作员认知负荷识别整体流程

下载: 全尺寸图片幻灯片

图 2 MATB II实验任务

下载: 全尺寸图片幻灯片

图 3 单个被试准确性

下载: 全尺寸图片幻灯片

图 4 不同方法识别性能

下载: 全尺寸图片幻灯片

图 5 不同方法识别性能

下载: 全尺寸图片幻灯片

图 6 被试数据分布

下载: 全尺寸图片幻灯片

表 1 特征提取器的结构

卷积层	类型	滤波器	核	输出
1	输入			(C, T, 1)
	卷积层	F₁=40	(1, K₁=25)	(C,T–K₁+1, F₁)
	批处理层
	ELU激活层
2	卷积层	F₂=40	(K₂=C, 1)	(1, T–K₁+1, F₂)
	批处理层
	平方激活层
	平均池化层	窗口(1, 75)	步长(1, 15)	(1,(T–K₁+1–75)// 15, F₂)
	对数激活层
	丢弃层
	铺平			(T–K₁+1–75)//15*F₂

下载: 导出CSV

表 2 每类F1和总体指标

方法	数据集1							数据集2
	每类F1 (%)			总体				每类 F1 (%)			总体
	低	中	高	ACC	MF1	K	Mgm	低	中	高	ACC	MF1	K	Mgm
SVM	60.51	41.90	52.62	51.86	51.68	0.28	62.57	56.93	34.33	50.44	47.77	47.23	0.22	58.95
LDA	54.30	37.93	45.58	45.95	45.94	0.19	57.83	49.05	36.95	46.21	44.06	44.07	0.16	56.30
KNN	50.43	39.07	32.74	41.54	40.75	0.12	54.23	47.89	37.32	35.69	40.48	40.30	0.11	53.25
RG	56.54	33.85	51.30	48.07	47.23	0.22	59.15	56.07	25.06	49.23	45.80	43.45	0.19	57.05
TJM	66.39	42.87	59.04	56.81	56.10	0.35	66.38	61.23	34.62	57.21	52.21	51.02	0.28	62.42
LSTM	37.41	38.20	39.20	38.31	38.27	0.07	51.64	43.02	33.59	40.39	39.33	39.00	0.09	52.31
DeepCNN	68.27	36.34	58.33	56.75	54.31	0.35	66.75	64.86	34.57	56.72	53.93	52.05	0.31	64.09
EEGNet	66.87	39.29	54.75	54.85	53.64	0.32	64.75	65.95	34.35	57.79	54.26	52.70	0.31	64.00
CNN_SE	67.70	44.87	57.55	57.23	56.71	0.36	66.78	63.90	38.85	57.79	54.51	53.51	0.32	64.52
CNN_rank1	62.78	47.75	54.75	55.48	55.10	0.33	65.37	61.01	37.37	45.05	49.72	47.81	0.25	58.84
TCN	66.87	42.75	56.17	55.38	55.26	0.33	65.42	67.18	39.00	56.03	54.97	54.07	0.32	64.73
CNN_DG	70.80	49.99	60.54	60.60	60.44	0.41	69.63	66.80	42.63	58.29	56.23	55.91	0.34	66.01

下载: 导出CSV

表 3 CNN_DA方法的混淆矩阵和每类结果

数据集1									数据集2
	预测				每类结果(%)						预测			每类结果(%)
真实		低	中	高	pre	sen	spe	F1	真实		低	中	高	pre	sen	spe	F1
	低	1589	435	230	71.10	70.50	85.49	70.80		低	1503	543	219	67.25	66.36	83.51	66.80
	中	379	1072	603	47.96	52.19	74.99	49.99		中	433	898	647	40.18	45.40	71.72	42.63
	高	267	728	1402	62.73	58.49	80.66	60.54		高	299	794	1369	61.25	55.61	79.59	58.29

下载: 导出CSV

参考文献(33)

[1]	OLUWAFEMI F A, ABDELBAKI R, LAI J C Y, et al. A review of astronaut mental health in manned missions: Potential interventions for cognitive and mental health challenges[J]. Life Sciences in Space Research, 2021, 28: 26–31. doi: 10.1016/j.lssr.2020.12.002
[2]	WU E Q, CAO Zhengtao, XIONG Pengwen, et al. Brain-computer interface using brain power map and cognition detection network during flight[J]. IEEE/ASME Transactions on Mechatronics, 2022, 27(5): 3942–3952. doi: 10.1109/TMECH.2022.3148141
[3]	LIN Yi, DENG Linjie, CHEN Zhengmao, et al. A real-time ATC safety monitoring framework using a deep learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(11): 4572–4581. doi: 10.1109/TITS.2019.2940992
[4]	WICKENS C D. Multiple resources and performance prediction[J]. Theoretical Issues in Ergonomics Science, 2002, 3(2): 159–177. doi: 10.1080/14639220210123806
[5]	ZHOU Yueying, HUANG Shuo, XU Ziming, et al. Cognitive workload recognition using EEG signals and machine learning: A review[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022, 14(3): 799–818. doi: 10.1109/TCDS.2021.3090217
[6]	APPEL T, GERJETS P, HOFFMAN S, et al. Cross-task and cross-participant classification of cognitive load in an emergency simulation game[J]. IEEE Transactions on Affective Computing, 2023, 14(2): 1558–1571. doi: 10.1109/TAFFC.2021.3098237
[7]	DEBIE E, ROJAS R F, FIDOCK J, et al. Multimodal fusion for objective assessment of cognitive workload: A review[J]. IEEE Transactions on Cybernetics, 2021, 51(3): 1542–1555. doi: 10.1109/TCYB.2019.2939399
[8]	CHARLES R L and NIXON J. Measuring mental workload using physiological measures: A systematic review[J]. Applied Ergonomics, 2019, 74: 221–232. doi: 10.1016/j.apergo.2018.08.028
[9]	APPRIOU A, CICHOCKI A, and LOTTE F. Modern machine-learning algorithms: For classifying cognitive and affective states from electroencephalography signals[J]. IEEE Systems, Man, and Cybernetics Magazine, 2020, 6(3): 29–38. doi: 10.1109/MSMC.2020.2968638
[10]	WANG Ziheng, HOPE R M, WANG Zuoguan, et al. Cross-subject workload classification with a hierarchical Bayes model[J]. NeuroImage, 2012, 59(1): 64–69. doi: 10.1016/j.neuroimage.2011.07.094
[11]	ZHANG Jianhua, WANG Yongcun, and LI Sunan. Cross-subject mental workload classification using kernel spectral regression and transfer learning techniques[J]. Cognition, Technology & Work, 2017, 19(4): 587–605. doi: 10.1007/s10111-017-0425-3
[12]	BHOSALE S, CHAKRABORTY R, and KOPPARAPU S K. Calibration free meta learning based approach for subject independent EEG emotion recognition[J]. Biomedical Signal Processing and Control, 2022, 72: 103289. doi: 10.1016/j.bspc.2021.103289
[13]	WANG Shouyi, GWIZDKA J, and CHAOVALITWONGSE W A. Using wireless EEG signals to assess memory workload in the n-back task[J]. IEEE Transactions on Human-Machine Systems, 2016, 46(3): 424–435. doi: 10.1109/THMS.2015.2476818
[14]	APPRIOU A, CICHOCKI A, and LOTTE F. Towards robust neuroadaptive HCI: Exploring modern machine learning methods to estimate mental workload from EEG signals[C]. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, Canada, 2018: 1–6.
[15]	BASHIVAN P, RISH I, YEASIN M, et al. Learning representations from EEG with deep recurrent-convolutional neural networks[C]. The 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2015.
[16]	JIAO Zhicheng, GAO Xinbo, WANG Ying, et al. Deep convolutional neural networks for mental load classification based on EEG data[J]. Pattern Recognition, 2018, 76: 582–595. doi: 10.1016/j.patcog.2017.12.002
[17]	ZHOU Yueying, XU Ziming, NIU Yifan, et al. Cross-task cognitive workload recognition based on EEG and domain adaptation[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 50–60. doi: 10.1109/TNSRE.2022.3140456
[18]	JIMÉNEZ-GUARNEROS M and GÓMEZ-GIL P. Custom domain adaptation: A new method for cross-subject, EEG-based cognitive load recognition[J]. IEEE Signal Processing Letters, 2020, 27: 750–754. doi: 10.1109/LSP.2020.2989663
[19]	WANG Jindong, LAN Cuiling, LIU Chang, et al. Generalizing to unseen domains: A survey on domain generalization[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(8): 8052–8072. doi: 10.1109/TKDE.2022.3178128
[20]	SCHIRRMEISTER R T, SPRINGENBERG J T, FIEDERER L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization[J]. Human Brain Mapping, 2017, 38(11): 5391–5420. doi: 10.1002/hbm.23730
[21]	GANIN Y, USTINOVA E, AJAKAN H, et al. Domain-adversarial training of neural networks[J]. The Journal of Machine Learning Research, 2016, 17(1): 2096–2030. doi: 10.5555/2946645.2946704
[22]	HEFRON R G, BORGHETTI B J, CHRISTENSEN J C, et al. Deep long short-term memory structures model temporal dependencies improving cognitive workload estimation[J]. Pattern Recognition Letters, 2017, 94: 96–104. doi: 10.1016/j.patrec.2017.05.020
[23]	LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGnet: A compact convolutional neural network for EEG-based brain–computer interfaces[J]. Journal of Neural Engineering, 2018, 15(5): 056013. doi: 10.1088/1741-2552/aace8c
[24]	HU Jie, SHEN Li, and SUN Gang. Squeeze-and-excitation networks[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 7132–7141.
[25]	INGOLFSSON T M, HERSCHE M, WANG Xiaying, et al. EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces[C]. 2020 IEEE International Conference on Systems, Man, and Cybernetics, Toronto, Canada, 2020: 2958–2965.
[26]	ROY R N, HINSS M F, DARMET L, et al. Retrospective on the first passive brain-computer interface competition on cross-session workload estimation[J]. Frontiers in Neuroergonomics, 2022, 3: 838342. doi: 10.3389/fnrgo.2022.838342
[27]	CONGEDO M, BARACHANT A, and BHATIA R. Riemannian geometry for EEG-based brain-computer interfaces; A primer and a review[J]. Brain-Computer Interfaces, 2017, 4(3): 155–174. doi: 10.1080/2326263X.2017.1297192
[28]	LA TOUR T D, MOREAU T, JAS M, et al. Multivariate convolutional sparse coding for electromagnetic brain signals[C]. The 32nd Conference on Neural Information Processing Systems, Montréal, Canada, 2018: 3296–3306.
[29]	KIM H, YOON J, JEONG B, et al. Rank-1 convolutional neural network[J]. arXiv: 1808.04303, 2018.
[30]	Passive BCI hackathon - neuroergonomics conference 2021[EB/OL]. https://www.neuroergonomicsconference.um.ifi.lmu.de/pbci/.
[31]	SANTIAGO-ESPADA Y. The Multi-Attribute Task Battery II (MATB-II) Software for Human Performance and Workload Research: A User’s Guide[M]. Hampton, Virginia: National Aeronautics and Space Administration, Langley Research Center, 2011.
[32]	WOOLSON R F. Wilcoxon signed-rank test[M]. D'AGOSTINO R B, SULLIVAN L, and MASSARO J. Wiley Encyclopedia of Clinical Trials. John Wiley & Sons, Inc. , 2007: 1–3.
[33]	VAN DER MAATEN L and HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(86): 2579–2605.