Cross-operator Cognitive Workload Recognition Based on Convolutional Neural Network and Domain Generalization

ZHOU Yueying; GONG Peiliang; WANG Pengpai; WEN Xuyun; ZHANG Daoqiang

doi:10.11999/JEIT221491

Volume 45 Issue 8

Aug. 2023

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2023 > 45(8): 2796-2805

ZHOU Yueying, GONG Peiliang, WANG Pengpai, WEN Xuyun, ZHANG Daoqiang. Cross-operator Cognitive Workload Recognition Based on Convolutional Neural Network and Domain Generalization[J]. Journal of Electronics & Information Technology, 2023, 45(8): 2796-2805. doi: 10.11999/JEIT221491

Citation:

ZHOU Yueying, GONG Peiliang, WANG Pengpai, WEN Xuyun, ZHANG Daoqiang. Cross-operator Cognitive Workload Recognition Based on Convolutional Neural Network and Domain Generalization[J]. Journal of Electronics & Information Technology, 2023, 45(8): 2796-2805. doi: 10.11999/JEIT221491

Citation:

ZHOU Yueying, GONG Peiliang, WANG Pengpai, WEN Xuyun, ZHANG Daoqiang. Cross-operator Cognitive Workload Recognition Based on Convolutional Neural Network and Domain Generalization[J]. Journal of Electronics & Information Technology, 2023, 45(8): 2796-2805. doi: 10.11999/JEIT221491

PDF( 3563 KB)

Cross-operator Cognitive Workload Recognition Based on Convolutional Neural Network and Domain Generalization

doi: 10.11999/JEIT221491

1.
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
2.
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
3.
Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China

Funds: The National Natural Science Foundation of China (62136004, 61876082, 61732006), The National Key R&D Program of China (2018YFC2001600, 2018YFC2001602), The Fundamental Research Funds for the Central Universities (NP2022451)

Received Date: 2022-11-30
Rev Recd Date: 2023-05-23

Available Online: 2023-05-30

Publish Date: 2023-08-21

Abstract

Abstract

ElectroEncephaloGraphy (EEG)-based Cognitive Workload Recognition (CWR) is valuable for human-robot interaction systems and passive brain-computer interfaces. However, the none-stationary of EEG and the difference between subjects hinder the rapid application of cross-operator CWR, a realistic scenario. To deal with the above problem, a jointly shared feature optimization method based on the Convolutional Neural Network (CNN) and Domain Generalization (DG) is proposed, denoted as CNN_DG. The data of existing operators (source domains) is used to improve the CWR performance of unknown operators (target domain). It includes three modules: EEG feature extractor, label classifier, and domain generalizer. The EEG feature extractor learns the transferable shared knowledge representation between source domains. The label classifier learns further the deep representation and predicted the workload levels. By adversarial training with the feature extractor, the domain generalizer reduces the difference in source domain distribution and ensures further the sharing of learned features. Two three-categories cross-operator CWR experiments are conducted on the Multi-attribute Task Battery (MATB II) simulated flight competition datasets 1 and 2, and the model performance is verified by using leave-one-subject-out cross-validation. Experimental results showed the CNN_DG performed significantly better than comparing methods, indicating its effectiveness and generalization in the field of cross-operator CWR.
- Human-robot interaction,
- Cognitive workload,
- Cross-operator,
- Convolutional Neural Network (CNN),
- Domain Generalization (DG)

FullText(HTML)

References(33)

References

[1]	OLUWAFEMI F A, ABDELBAKI R, LAI J C Y, et al. A review of astronaut mental health in manned missions: Potential interventions for cognitive and mental health challenges[J]. Life Sciences in Space Research, 2021, 28: 26–31. doi: 10.1016/j.lssr.2020.12.002
[2]	WU E Q, CAO Zhengtao, XIONG Pengwen, et al. Brain-computer interface using brain power map and cognition detection network during flight[J]. IEEE/ASME Transactions on Mechatronics, 2022, 27(5): 3942–3952. doi: 10.1109/TMECH.2022.3148141
[3]	LIN Yi, DENG Linjie, CHEN Zhengmao, et al. A real-time ATC safety monitoring framework using a deep learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(11): 4572–4581. doi: 10.1109/TITS.2019.2940992
[4]	WICKENS C D. Multiple resources and performance prediction[J]. Theoretical Issues in Ergonomics Science, 2002, 3(2): 159–177. doi: 10.1080/14639220210123806
[5]	ZHOU Yueying, HUANG Shuo, XU Ziming, et al. Cognitive workload recognition using EEG signals and machine learning: A review[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022, 14(3): 799–818. doi: 10.1109/TCDS.2021.3090217
[6]	APPEL T, GERJETS P, HOFFMAN S, et al. Cross-task and cross-participant classification of cognitive load in an emergency simulation game[J]. IEEE Transactions on Affective Computing, 2023, 14(2): 1558–1571. doi: 10.1109/TAFFC.2021.3098237
[7]	DEBIE E, ROJAS R F, FIDOCK J, et al. Multimodal fusion for objective assessment of cognitive workload: A review[J]. IEEE Transactions on Cybernetics, 2021, 51(3): 1542–1555. doi: 10.1109/TCYB.2019.2939399
[8]	CHARLES R L and NIXON J. Measuring mental workload using physiological measures: A systematic review[J]. Applied Ergonomics, 2019, 74: 221–232. doi: 10.1016/j.apergo.2018.08.028
[9]	APPRIOU A, CICHOCKI A, and LOTTE F. Modern machine-learning algorithms: For classifying cognitive and affective states from electroencephalography signals[J]. IEEE Systems, Man, and Cybernetics Magazine, 2020, 6(3): 29–38. doi: 10.1109/MSMC.2020.2968638
[10]	WANG Ziheng, HOPE R M, WANG Zuoguan, et al. Cross-subject workload classification with a hierarchical Bayes model[J]. NeuroImage, 2012, 59(1): 64–69. doi: 10.1016/j.neuroimage.2011.07.094
[11]	ZHANG Jianhua, WANG Yongcun, and LI Sunan. Cross-subject mental workload classification using kernel spectral regression and transfer learning techniques[J]. Cognition, Technology & Work, 2017, 19(4): 587–605. doi: 10.1007/s10111-017-0425-3
[12]	BHOSALE S, CHAKRABORTY R, and KOPPARAPU S K. Calibration free meta learning based approach for subject independent EEG emotion recognition[J]. Biomedical Signal Processing and Control, 2022, 72: 103289. doi: 10.1016/j.bspc.2021.103289
[13]	WANG Shouyi, GWIZDKA J, and CHAOVALITWONGSE W A. Using wireless EEG signals to assess memory workload in the n-back task[J]. IEEE Transactions on Human-Machine Systems, 2016, 46(3): 424–435. doi: 10.1109/THMS.2015.2476818
[14]	APPRIOU A, CICHOCKI A, and LOTTE F. Towards robust neuroadaptive HCI: Exploring modern machine learning methods to estimate mental workload from EEG signals[C]. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, Canada, 2018: 1–6.
[15]	BASHIVAN P, RISH I, YEASIN M, et al. Learning representations from EEG with deep recurrent-convolutional neural networks[C]. The 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2015.
[16]	JIAO Zhicheng, GAO Xinbo, WANG Ying, et al. Deep convolutional neural networks for mental load classification based on EEG data[J]. Pattern Recognition, 2018, 76: 582–595. doi: 10.1016/j.patcog.2017.12.002
[17]	ZHOU Yueying, XU Ziming, NIU Yifan, et al. Cross-task cognitive workload recognition based on EEG and domain adaptation[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 50–60. doi: 10.1109/TNSRE.2022.3140456
[18]	JIMÉNEZ-GUARNEROS M and GÓMEZ-GIL P. Custom domain adaptation: A new method for cross-subject, EEG-based cognitive load recognition[J]. IEEE Signal Processing Letters, 2020, 27: 750–754. doi: 10.1109/LSP.2020.2989663
[19]	WANG Jindong, LAN Cuiling, LIU Chang, et al. Generalizing to unseen domains: A survey on domain generalization[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(8): 8052–8072. doi: 10.1109/TKDE.2022.3178128
[20]	SCHIRRMEISTER R T, SPRINGENBERG J T, FIEDERER L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization[J]. Human Brain Mapping, 2017, 38(11): 5391–5420. doi: 10.1002/hbm.23730
[21]	GANIN Y, USTINOVA E, AJAKAN H, et al. Domain-adversarial training of neural networks[J]. The Journal of Machine Learning Research, 2016, 17(1): 2096–2030. doi: 10.5555/2946645.2946704
[22]	HEFRON R G, BORGHETTI B J, CHRISTENSEN J C, et al. Deep long short-term memory structures model temporal dependencies improving cognitive workload estimation[J]. Pattern Recognition Letters, 2017, 94: 96–104. doi: 10.1016/j.patrec.2017.05.020
[23]	LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGnet: A compact convolutional neural network for EEG-based brain–computer interfaces[J]. Journal of Neural Engineering, 2018, 15(5): 056013. doi: 10.1088/1741-2552/aace8c
[24]	HU Jie, SHEN Li, and SUN Gang. Squeeze-and-excitation networks[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 7132–7141.
[25]	INGOLFSSON T M, HERSCHE M, WANG Xiaying, et al. EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces[C]. 2020 IEEE International Conference on Systems, Man, and Cybernetics, Toronto, Canada, 2020: 2958–2965.
[26]	ROY R N, HINSS M F, DARMET L, et al. Retrospective on the first passive brain-computer interface competition on cross-session workload estimation[J]. Frontiers in Neuroergonomics, 2022, 3: 838342. doi: 10.3389/fnrgo.2022.838342
[27]	CONGEDO M, BARACHANT A, and BHATIA R. Riemannian geometry for EEG-based brain-computer interfaces; A primer and a review[J]. Brain-Computer Interfaces, 2017, 4(3): 155–174. doi: 10.1080/2326263X.2017.1297192
[28]	LA TOUR T D, MOREAU T, JAS M, et al. Multivariate convolutional sparse coding for electromagnetic brain signals[C]. The 32nd Conference on Neural Information Processing Systems, Montréal, Canada, 2018: 3296–3306.
[29]	KIM H, YOON J, JEONG B, et al. Rank-1 convolutional neural network[J]. arXiv: 1808.04303, 2018.
[30]	Passive BCI hackathon - neuroergonomics conference 2021[EB/OL]. https://www.neuroergonomicsconference.um.ifi.lmu.de/pbci/.
[31]	SANTIAGO-ESPADA Y. The Multi-Attribute Task Battery II (MATB-II) Software for Human Performance and Workload Research: A User’s Guide[M]. Hampton, Virginia: National Aeronautics and Space Administration, Langley Research Center, 2011.
[32]	WOOLSON R F. Wilcoxon signed-rank test[M]. D'AGOSTINO R B, SULLIVAN L, and MASSARO J. Wiley Encyclopedia of Clinical Trials. John Wiley & Sons, Inc. , 2007: 1–3.
[33]	VAN DER MAATEN L and HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(86): 2579–2605.