WorkloadGPT: A Large Language Model Approach to Real-Time Detection of Pilot Workload
Source abstract
Abstract
The occurrence of flight risks and accidents is closely related to pilot workload. Effective detection of pilot workload has been a key research area in the aviation industry. However, traditional methods for detecting pilot workload have several shortcomings: firstly, the collection of metrics via contact-based devices can interfere with pilots; secondly, real-time detection of pilot workload is challenging, making it difficult to capture sudden increases in workload; thirdly, the detection accuracy of these models is limited; fourthly, the models lack cross-pilot generalization. To address these challenges, this study proposes a large language model, WorkloadGPT, which utilizes low-interference indicators: eye movement and seat pressure. Specifically, features are extracted in 10 s time windows and input into WorkloadGPT for classification into low, medium, and high workload categories. Additionally, this article presents the design of an appropriate text template to serialize the tabular feature dataset into natural language, incorporating individual difference prompts during instance construction to enhance cross-pilot generalization. Finally, the LoRA algorithm was used to fine-tune the pre-trained large language model ChatGLM3-6B, resulting in WorkloadGPT. During the training process of WorkloadGPT, the GAN-Ensemble algorithm was employed to augment the experimental raw data, constructing a realistic and robust extended dataset for model training. The results show that WorkloadGPT achieved a classification accuracy of 87.3%, with a cross-pilot standard deviation of only 2.1% and a response time of just 1.76 s, overall outperforming existing studies in terms of accuracy, real-time performance, and cross-pilot generalization capability, thereby providing a solid foundation for enhancing flight safety.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human-Automation Interaction and Safety · Occupational Health and Safety Research · Risk and Safety Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of Road and Traffic Engineering of Ministry of Education, Department of Transportation Engineering, Tongji University, Shanghai 201804, China
College of Engineering, Hohai University, Nanjing 210024, China
Funding
National Natural Science Foundation Project
References (81)
Show source-provided references
1. Situation awareness and workload in aviation10.1111/1467-8721.00184 ↗
2. Cockpit information required by advanced fighter pilots for displays in taxiing take-off
3. Measurement and evaluation of mental workload for aircraft cockpit display interface
4. Kale, U., Rohács, J., and Rohács, D. (2020). Operators’ load monitoring and management. Sensors, 20.10.3390/s20174665 ↗
5. A comparative analysis of signal processing and classification methods for different applications based on EEG signals10.1016/j.bbe.2020.02.002 ↗
6. Driver cognitive loads based on multi-dimensional information feature analysis
7. Electroencephalogram-based cognitive load level classification using wavelet decomposition and support vector machine10.1080/2326263X.2022.2109855 ↗
8. Salimi, N., Barlow, M., and Lakshika, E. (2019, January 6–9). Mental Workload Classification Using Short Duration EEG Data: An Ensemble Approach Based on Individual Channels. Proceedings of the 2019 IEEE Symposium Series on Computational Intelligence (SSCI), Xiamen, China.10.1109/SSCI44817.2019.9003141 ↗
9. Khan, M.J., Hong, M.J., and Hong, K.S. (2014). Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface. Front. Hum. Neurosci., 8.10.3389/fnhum.2014.00244 ↗
10. Enhanced performance by a hybrid NIRS–EEG brain computer interface10.1016/j.neuroimage.2011.07.084 ↗
11. Neural network classification of brain hemodynamic responses from four mental tasks10.1080/15599612.2011.633209 ↗
12. Hennrich, J., Herff, C., Heger, D., and Schultz, T. (2015, January 25–29). Investigating deep learning for fNIRS based BCI. Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy.10.1109/EMBC.2015.7318984 ↗
13. Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: Three-class classification of rest, right-, and left-hand motor execution
14. Ma, L., Zhang, L., Wang, L., Xu, M., Qi, H., Wan, B., Ming, D., and Hu, Y. (2012, January 2–4). A hybrid brain-computer interface combining the EEG and NIRS. Proceedings of the 2012 IEEE International Conference on Virtual Environments Human-Computer Interfaces and Measurement Systems (VECIMS), Tianjin, China.10.1109/VECIMS.2012.6273214 ↗
15. Lee, M.H., Fazli, S., Mehnert, J., and Lee, S.W. (2014, January 17–19). Hybrid brain-computer interface based on EEG and NIRS modalities. Proceedings of the 2014 International Winter Workshop on Brain-Computer Interface (BCI), Jeongseon, Republic of Korea.10.1109/iww-BCI.2014.6782577 ↗
16. A survey on video action recognition in sports: Datasets, methods and applications10.1109/TMM.2022.3232034 ↗
17. Ru, Y., Wei, Z., An, G., and Chen, H. (2024). Combining data augmentation and deep learning for improved epilepsy detection. Front. Neurol., 15.10.3389/fneur.2024.1378076 ↗
18. Multimodal behavioral and physiological signals as indicators of cognitive load
19. Rashid, H., Tanveer, M.A., and Khan, H.A. (2019, January 23–27). Skin lesion classification using GAN based data augmentation. Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany.10.1109/EMBC.2019.8857905 ↗
20. Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., and Zhong, T. (2024). Understanding LLMs: A comprehensive overview from training to inference. arXiv.10.2139/ssrn.4706201 ↗
21. Functional Near-Infrared Spectroscopy in the Evaluation of Urban Rail Transit Drivers’ Mental Workload under Simulated Driving Conditions10.1080/00140139.2018.1535093 ↗
22. Mental Workload Measurement for Emergency Operating Procedures in Digital Nuclear Power Plants10.1080/00140139.2013.790483 ↗
23. Profiling Cognitive Workload in an Unmanned Vehicle Control Task with Cognitive Models and Physiological Metrics10.1080/08995605.2022.2130673 ↗
24. Mental Workload since 1979
25. STEW: Simultaneous Task EEG Workload Data Set10.1109/TNSRE.2018.2872924 ↗
26. Classification of Pilots’ Mental States Using a Multimodal Deep Learning Network10.1016/j.bbe.2019.12.002 ↗
27. Classification of Driver Cognitive Load: Exploring the Benefits of Fusing Eye-Tracking and Physiological Measures10.1177/03611981221090937 ↗
28. Rahman, H., Ahmed, M., Barua, S., and Begum, S. (2020). Non-Contact-Based Driver’s Cognitive Load Classification Using Physiological and Vehicular Parameters. Biomed. Signal Process. Control, 55.10.1016/j.bspc.2019.101634 ↗
29. Zhang, X., Sun, Y., Qiu, Z., Bao, J., and Zhang, Y. (2019). Adaptive Neuro-Fuzzy Fusion of Multi-Sensor Data for Monitoring a Pilot’s Workload Condition. Sensors, 19.10.3390/s19163629 ↗
30. Facilitating the Child–Robot Interaction by Endowing the Robot with the Capability of Understanding the Child Engagement: The Case of Mio Amico Robot10.1007/s12369-020-00661-w ↗
31. Hyönä, J., Radach, R., and Deubel, H. (2003). Commentary on Section 4—Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises. The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, Elsevier.
32. A Sensing Chair Using Pressure Distribution Sensors10.1109/3516.951364 ↗
33. Method for the Analysis of Posture and Interface Pressure of Car Drivers10.1016/S0003-6870(02)00069-8 ↗
34. Lantoine, P., Lecocq, M., Bougard, C., Dousset, E., Marqueste, T., Bourdin, C., Allègre, J.M., Bauvineau, L., and Mesure, S. (2021). Car Seat Impact on Driver’s Sitting Behavior and Perceived Discomfort during Prolonged Real Driving on Varied Road Types. PLoS ONE, 16.10.1371/journal.pone.0259934 ↗
35. Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., and Van de Weijer, J. (2011). Eye Tracking: A Comprehensive Guide to Methods and Measures, OUP Oxford.
36. Eye Movements in Reading and Information Processing: 20 Years of Research10.1037/0033-2909.124.3.372 ↗
37. Saccadic Peak Velocity Sensitivity to Variations in Mental Workload10.3357/ASEM.2579.2010 ↗
38. The Main Sequence, a Tool for Studying Human Eye Movements10.1016/0025-5564(75)90075-9 ↗
39. Blink Rate: A Possible Measure of Fatigue10.1177/001872089403600209 ↗
40. Task-Evoked Pupillary Responses, Processing Load, and the Structure of Processing Resources10.1037/0033-2909.91.2.276 ↗
41. Vibrotactile In-Vehicle Navigation System10.1016/j.trf.2004.09.003 ↗
42. Zhang, Q., Gui, T., Zheng, R., and Huang, X. (2024, May 17). The Theory and Practice of Large Language Models. Available online: https://intro-llm.github.io/.
43. Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D. (2022). TabLLM: Few-Shot Classification of Tabular Data with Large Language Models. arXiv, Available online: https://arxiv.org/abs/2210.10723.
44. Fang, X., Xu, W., Tan, F.A., Zhang, J., Hu, Z., Qi, Y., Nickleach, S., Socolinsky, D., Sengamedu, S., and Faloutsos, C. (2024). Large Language Models (LLMs) on Tabular Data: Prediction, Generation, and Understanding—A Survey. arXiv, Available online: https://arxiv.org/abs/2402.17944.
45. Zhang, Y., Wang, M., and Ren, C. (2024). Pushing the Limit of LLM Capacity for Text Classification. arXiv, Available online: https://arxiv.org/abs/2402.07470.
46. Herzig, J., Nowak, P.K., Müller, T., Piccinno, F., and Eisenschlos, J.M. (2020). TaPas: Weakly Supervised Table Parsing via Pre-Training. arXiv.10.18653/v1/2020.acl-main.398 ↗
47. Nassar, A., Livathinos, N., Lysak, M., and Staar, P. (2022, January 18–24). Tableformer: Table Structure Understanding with Transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00457 ↗
48. Turl: Table Understanding through Representation Learning10.1145/3542700.3542709 ↗
49. Liu, Q., Chen, B., Guo, J., Ziyadi, M., Lin, Z., Chen, W., and Lou, J.G. (2021). TAPEX: Table Pre-Training via Learning a Neural SQL Executor. arXiv.
50. Iida, H., Thai, D., Manjunatha, V., and Iyyer, M. (2021). Tabbie: Pretrained Representations of Tabular Data. arXiv.10.18653/v1/2021.naacl-main.270 ↗
51. Gong, H., Sun, Y., Feng, X., Qin, B., Bi, W., Liu, X., and Liu, T. (2020, January 8–13). TableGPT: Few-shot table-to-text generation with table structure reconstruction and content matching. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.179 ↗
52. Xu, X. (2023). Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data. arXiv, Available online: https://arxiv.org/abs/2307.14385.
53. Ma, Z., Guo, H., Chen, J., Peng, G., Cao, Z., Ma, Y., and Gong, Y.-J. (2024). LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation. arXiv, Available online: https://arxiv.org/abs/2403.01131.
54. Tribes, C., Benarroch-Lelong, S., Lu, P., and Kobyzev, I. (2024). Hyperparameter Optimization for Large Language Model Instruction-Tuning. arXiv, Available online: https://arxiv.org/abs/2312.00949.
55. Detecting Congestive Heart Failure by Extracting Multimodal Features with Synthetic Minority Oversampling Technique (SMOTE) for Imbalanced Data Using Robust Machine Learning Techniques10.1080/17455030.2020.1810364 ↗
56. Generative Adversarial Networks (GANs) Challenges, Solutions, and Future Directions
57. SMOTE: Synthetic Minority Over-sampling Technique10.1613/jair.953 ↗
58. Xu, L., Skoularidou, M., Cuesta-Infante, A., and Veeramachaneni, K. (2019). Modeling Tabular Data Using Conditional GAN. arXiv.
59. Zhang, C., Hou, Y., Chen, K., Cao, S., Fan, G., and Liu, J. (2023, January 7–14). Quality-aware self-training on differentiable synthesis of rare relational data. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i5.25811 ↗
60. Hajikhani, A., and Cole, C. (2024). A critical review of large language models: Sensitivity, bias, and the path toward specialized AI. Quant. Sci. Stud., 1–22.10.1162/qss_a_00310 ↗
61. Loya, M., Sinha, D.A., and Futrell, R. (2023). Exploring the Sensitivity of LLMs’ Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters. arXiv.10.18653/v1/2023.findings-emnlp.241 ↗
62. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv.
63. Sui, Y., Zhou, M., Zhou, M., Han, S., and Zhang, D. (2024, January 4–8). Table meets llm: Can large language models understand structured table data? A benchmark and empirical study. Proceedings of the 17th ACM International Conference on Web Search and Data Mining, Merida, Mexico.10.1145/3616855.3635752 ↗
64. System identification of an unmanned aerial vehicle with actuated wingtips10.1007/s10846-022-01599-z ↗
65. Zeng, G., Jia, H., Li, J., Pan, S., and Zheng, L. (2022, January 23–25). Design of UAV 3D visual simulation system based on X-plane. Proceedings of the International Conference on Autonomous Unmanned Systems, Xi’an, China.10.1007/978-981-99-0479-2_108 ↗
66. Garcia, R., and Barnes, L. (2009, January 8–10). Multi-uav simulator utilizing x-plane. Proceedings of the 2nd International Symposium on UAVs, Reno, NV, USA.10.1007/978-90-481-8764-5_20 ↗
67. Development of selection methods for civil aviation student pilots. Part I: Comparison of aptitude testscores between China and Germany
68. Popov, A., and Ivanko, K. (2024). Introduction to biomedical signals and biomedical imaging. Advances in Artificial Intelligence, Academic Press.10.1016/B978-0-443-19073-5.00013-6 ↗
69. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis10.1016/j.jneumeth.2003.10.009 ↗
70. Wavelet based machine learning models for classification of human emotions using EEG signal10.1016/j.measen.2022.100554 ↗
71. Guo, H. (2020, January 7–12). Nonlinear mixup: Out-of-manifold data augmentation for text classification. Proceedings of the AAAI Conference on Artificial Intelligence, Hilton, NY, USA.10.1609/aaai.v34i04.5822 ↗
72. Zhang, H., Cisse, M., Dauphin, Y.N., and Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. arXiv.
73. Maron, H., Ben-Hamu, H., Serviansky, H., and Lipman, Y. (2019). Provably powerful graph networks. Advances in Neural Information Processing Systems, MIT Press.
74. Cross-operator Cognitive Workload Recognition Based on Convolutional Neural Network and Domain Generalization
75. Chen, J., Zhang, Q., Cheng, L., Gao, X., and Ding, L. (2019, January 16–19). A cognitive load assessment method considering individual differences in eye movement data. Proceedings of the 2019 IEEE 15th International Conference on Control and Automation (ICCA), Edinburgh, UK.10.1109/ICCA.2019.8899595 ↗
76. Nonlinear analysis of electrodermal activity signals for healthy subjects and patients with chronic obstructive pulmonary disease10.1007/s13246-018-0649-4 ↗
77. Kingma, D.P. (2013). Auto-Encoding Variational Bayes. arXiv, Available online: https://arxiv.org/abs/1312.6114.
78. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems, MIT Press.
79. Overcoming Catastrophic Forgetting in Neural Networks10.1073/pnas.1611835114 ↗
80. Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. (2018, January 8–14). Memory Aware Synapses: Learning What (Not) to Forget. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01219-9_9 ↗
81. Jiang, G., Li, Z., Jiang, C., Xue, S., Zhou, J., Song, L., Lian, D., and Wei, Y. (2024). Interpretable Catastrophic Forgetting of Large Language Model Fine-Tuning via Instruction Vector. arXiv, Available online: https://arxiv.org/abs/2406.12227.