journal article·2022-02-22Open access

Recognition of the Mental Workloads of Pilots in the Cockpit Using EEG Signals

Aura Hernández-Sabaté, José Yauri, Pau Folch, Miquel Àngel Piera, Debora Gil

Applied Sciences· Volume 12· Issue 5

Abstract

The commercial flightdeck is a naturally multi-tasking work environment, one in which interruptions are frequent come in various forms, contributing in many cases to aviation incident reports. Automatic characterization of pilots’ workloads is essential to preventing these kind of incidents. In addition, minimizing the physiological sensor network as much as possible remains both a challenge and a requirement. Electroencephalogram (EEG) signals have shown high correlations with specific cognitive and mental states, such as workload. However, there is not enough evidence in the literature to validate how well models generalize in cases of new subjects performing tasks with workloads similar to the ones included during the model’s training. In this paper, we propose a convolutional neural network to classify EEG features across different mental workloads in a continuous performance task test that partly measures working memory and working memory capacity. Our model is valid at the general population level and it is able to transfer task learning to pilot mental workload recognition in a simulated operational environment.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Human-Automation Interaction and Safety · EEG and Brain-Computer Interfaces · Healthcare Technology and Patient Monitoring

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Computer Vision Center, Universitat Autònoma de Barcelona, Bellaterra, 08193 Barcelona, Spain

Departament de Ciències de la Computació, Universitat Autònoma de Barcelona, Bellaterra, 08193 Barcelona, Spain

Aslogic, Parc de Recerca UAB, Bellaterra, 08193 Barcelona, Spain

Telecommunications and Systems Engineering Department, Universitat Autònoma de Barcelona, Bellaterra, 08193 Barcelona, Spain

Funding

Ministry of Economy, Industry and Competitiveness · Clean Sky · Agency for Administration of University and Research · Government of Catalonia

References (45)

Show source-provided references
1. Latorella, K.A. (1999). Investigating Interruptions: Implications for Flightdeck Performance, NASA.
2. Individual differences in working-memory capacity and task resumption following interruptions10.1037/xlm0000251 ↗
3. Long-term working memory and interrupting messages in human–computer interaction10.1080/01449290310001644859 ↗
4. Coping with competing demands: Interruption and the type A pattern10.1037/0021-9010.73.4.621 ↗
5. Interference between switched tasks10.1080/00140139208967795 ↗
6. Learning Spatial-Spectral-Temporal EEG Features With Recurrent 3D Convolutional Neural Networks for Cross-Task Mental Workload Assessment10.1109/TNSRE.2018.2884641 ↗
7. Li, D., Wang, X., Menassa, C.C., and Kamat, V.R. (2020). Understanding the impact of building thermal environments on occupants’ comfort and mental workload demand through human physiological sensing. Start-Up Creation, Elsevier.10.1016/B978-0-12-819946-6.00012-6 ↗
8. Combining time and intensity effects in assessing operator information-processing load10.1518/001872097778940597 ↗
9. Electrocardiographic features for the measurement of drivers’ mental workload10.1016/j.apergo.2016.12.015 ↗
10. Classification of pilots’ mental states using a multimodal deep learning network10.1016/j.bbe.2019.12.002 ↗
11. Spectral and Temporal Feature Learning with Two-Stream Neural Networks for Mental Workload Assessment10.1109/TNSRE.2019.2913400 ↗
12. Continuous EEG Decoding of Pilots’ Mental States Using Multiple Feature Block-Based Convolutional Neural Network10.1109/ACCESS.2020.3006907 ↗
13. Pilots’ fatigue status recognition using deep contractive autoencoder network10.1109/TIM.2018.2885608 ↗
14. Mental workload in air traffic control: An index constructed from field tests
15. Mental Workload, Task Demand and Driving Performance: What Relation?10.1016/j.sbspro.2014.12.212 ↗
16. NASA-task load index (NASA-TLX); 20 years later10.1177/154193120605000909 ↗
17. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness10.1016/j.neubiorev.2012.10.003 ↗
18. Hart, S.G., and Staveland, L.E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, Elsevier.10.1016/S0166-4115(08)62386-9 ↗
19. Situation awareness and workload in aviation10.1111/1467-8721.00184 ↗
20. Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs10.1518/155534308X284417 ↗
21. Application of heart rate variability in evaluation of mental workload
22. Stanton, N., Salmon, P.M., and Rafferty, L.A. (2013). Human Factors Methods: A Practical Guide for Engineering and Design, Ashgate Publishing, Ltd.
23. Jang, E.H., Park, B.J., Kim, S.H., Chung, M.A., Park, M.S., and Sohn, J.H. (March, January 24). Classification of human emotions from physiological signals using machine learning algorithms. Proceedings of the Sixth International Conference on Advances in Computer-Human Interactions 2013 (ACHI 2013), Nice, France.
24. Rim, B., Sung, N.J., Min, S., and Hong, M. (2020). Deep learning in physiological signal data: A survey. Sensors, 20.10.3390/s20040969 ↗
25. Ziegler, M.D., Russell, B.A., Kraft, A.E., Krein, M., Russo, J., and Casebeer, W.D. (2019). Computational Models for Near-real-time Performance Predictions Based on Physiological Measures of Workload. Neuroergonomics, Elsevier.10.1016/B978-0-12-811926-6.00019-1 ↗
26. Secerbegovic, A., Ibric, S., Nisic, J., Suljanovic, N., and Mujcic, A. (2017). Mental workload vs. stress differentiation using single-channel EEG. CMBEBIH 2017, Springer.10.1007/978-981-10-4166-2_78 ↗
27. Deep Learning for Time Series Classification: A Review10.1007/s10618-019-00619-1 ↗
28. Deep learning for healthcare applications based on physiological signals: A review10.1016/j.cmpb.2018.04.005 ↗
29. An investigation of deep learning models for EEG-based emotion recognition10.3389/fnins.2020.622759 ↗
30. A Novel Method of motor imagery classification using eeg signal10.1016/j.artmed.2019.101787 ↗
31. SeizureNet: A model for robust detection of epileptic seizures based on convolutional neural network10.1049/ccs.2020.0011 ↗
32. Automated multi-model deep neural network for sleep stage scoring with unfiltered clinical data10.1007/s11325-019-02008-w ↗
33. Ayaz, H. (2020). Convolutional Neural Network for Hybrid fNIRS-EEG Mental Workload Classification. Advances in Neuroergonomics and Cognitive Engineering, Springer International Publishing.10.1007/978-3-030-20473-0 ↗
34. Multilevel feature fusion with 3d convolutional neural network for eeg-based workload estimation10.1109/ACCESS.2020.2966834 ↗
35. Ziegler, M.D., Kraft, A., Krein, M., Lo, L.C., Hatfield, B., Casebeer, W., and Russell, B. (2016). Sensing and assessing cognitive workload across multiple tasks. International Conference on Augmented Cognition, Springer.10.1007/978-3-319-39955-3_41 ↗
36. Improving fluid intelligence with training on working memory10.1073/pnas.0801268105 ↗
37. A cross-disciplinary comparison of multimodal data fusion approaches and applications: Accelerating learning through trans-disciplinary information sharing10.1016/j.eswa.2020.113885 ↗
38. Yauri, J., Hernández-Sabaté, A., Folch, P., and Gil, D. (2021). Mental Workload Detection Based on EEG Analysis. Artificial Intelligence Research and Development, IOS Press.10.3233/FAIA210144 ↗
39. A socio-technical holistic agent based model to assess cockpit supporting tools performance variability10.1016/j.ifacol.2019.09.128 ↗
40. Emotiv (2021). EMOTIV EPOC+ 14-Channel Wireless EEG Headset, Emotiv.
41. Prestimulus Theta Activity Predicts Correct Source Memory Retrieval10.1073/pnas.1014528108 ↗
42. Wasserman, L. (2010). All of Statistics: A Concise Course in Statistical Inference, Springer.
43. Classification of cross task cognitive workload using deep recurrent network with modelling of temporal dynamics10.1016/j.bspc.2021.103070 ↗
44. Kingma, D.P., and Ba, J. (2014, January 14–16). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015-Conference Track Proceedings, Banff, AB, Canada.
45. EEG-based detection of mental workload level and stress: The effect of variation in each state on classification of the other10.1088/1741-2552/abbc27 ↗