Multimodal Emotion Recognition for Seafarers: A Framework Integrating Improved D-S Theory and Calibration: A Case Study of a Real Navigation Experiment
Source abstract
Abstract
The influence of seafarers’ emotions on work performance can lead to severe marine accidents. However, research on emotion recognition (ER) of seafarers remains insufficient, and existing studies only deploy single models and disregard the model’s uncertainty, which might lead to unreliable recognition. In this paper, a novel fusion framework for seafarer ER is proposed. Firstly, feature-level fusion using Electroencephalogram (EEG) and navigation data collected in a real navigation environment was conducted. Then, calibration is employed to mitigate the uncertainty of the outcomes. Secondly, a weight combination strategy for decision fusion was designed. Finally, we conduct a series of evaluations of the proposed model. The results showed that the average recognition performance across the three emotional dimensions, as measured by accuracy, precision, recall, and F1 score, reaches 85.14%, 84.43%, 86.27%, and 85.33%, respectively. The results demonstrate that the use of physiological and navigation data can effectively identify seafarers’ emotional states. Additionally, the fusion model compensates for the uncertainty of single models and enhances the performance of ER for seafarers, which provides a feasible path for the ER of seafarers. The findings of this study can be used to promptly identify the emotional state of seafarers and develop early warnings for bridge systems for shipping companies and help inform policy-making on human factors to enhance maritime safety.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Maritime Navigation and Safety · Marine and Coastal Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
State Key Laboratory of Maritime Technology and Safety, School of Transportation and Logistics Engineering, Wuhan University of Technology, Wuhan 430063, China
School of Transportation and Logistics Engineering, Wuhan University of Technology, Wuhan 430063, China
Funding
Science and Technology Innovation and Demonstration Project of Department of Transport of Yunnan Province · Hubei International Science and Technology Cooperation Project · Self-Innovation Foundation of State Key Laboratory of Maritime Technology and Safety · Research and development Project of China COSCO Shipping Corporation Limited
References (58)
Show source-provided references
1. Research on human factors cause chain of ship accidents based on multidimensional association rules10.1016/j.oceaneng.2020.107717 ↗
2. Predictive Power of Inspection Outcomes for Future Shipping Accidents—An Empirical Appraisal with Special Attention for Human Factor Aspects10.1080/03088839.2018.1440441 ↗
3. Searching for the origins of the myth 80% human error impact on maritime safety10.1016/j.ress.2021.107942 ↗
4. (2025, August 17). AGCS, Safety and Shipping Review 2022. Retrieved May 10, 2022 [EB/OL]. Available online: https://www.allianz.com/en/mediacenter/news/studies/220510_Allianz-AGCS-PressRelease-Safety-Shipping-Review-2022.html.
5. Fan, S., Yan, X., Zhang, J., and Wang, J. (2017, January 8–10). A review on human factors in maritime transportation using seafarers’ physiological data. Proceedings of the 2017 4th International Conference on Transportation Information and Safety (ICTIS), Banff, AB, Canada.10.1109/ICTIS.2017.8047751 ↗
6. Screening for anxiety, depression and poor psychological well-being in Spanish seafarers: An empirical study of the cut-off points on three measures of psychological functioning10.1016/j.oceaneng.2024.118572 ↗
7. Stereotype threat as a determinant of burnout or work engagement: Mediating role of positive and negative emotions10.1080/10803548.2015.1017939 ↗
8. Dzedzickis, A., Kaklauskas, A., and Bucinskas, V. (2020). Human Emotion Recognition: Review of Sensors and Methods. Sensors, 20.10.3390/s20030592 ↗
9. SAM: The Self-Assessment Manikin: An Efficient Cross-Cultural Measurement of Emotional Response10.1080/00218499.1995.12466497 ↗
10. Affective computing—A rationale for measuring mood with mouse and keyboard10.1080/10803548.2003.11076589 ↗
11. Liu, Y., and Sourina, O. (2014). Real-Time Subject-Dependent EEG-Based Emotion Recognition Algorithm. Transactions on Computational Science XXIII, Springer.10.1007/978-3-662-43790-2_11 ↗
12. Hou, X., Liu, Y., Lim, W.L., Lan, Z., Sourina, O., Mueller-Wittig, W., and Wang, L. (2016). CogniMeter: EEG-Based Brain States Monitoring. Transactions on Computational Science XXVIII, Springer. Lecture Notes in Computer Science.10.1007/978-3-662-53090-0_6 ↗
13. Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty10.1109/TSMCA.2011.2116000 ↗
14. Effects of seafarers’ emotion on human performance using bridge simulation10.1016/j.oceaneng.2018.10.021 ↗
15. E-navigation: Integrating physiological readings within the ship’ s bridge infrastructure10.1016/j.trpro.2025.02.045 ↗
16. AIS data repair model based on generative adversarial network10.1016/j.ress.2023.109572 ↗
17. Risk Prediction for Ship Encounter Situation Awareness Using Long Short-Term Memory Based Deep Learning on Intership Behaviors10.1155/2020/8897700 ↗
18. Extracting vessel speed based on machine learning and drone images during ship traffic flow prediction10.1155/2022/3048611 ↗
19. Enhancing emotion recognition using multimodal fusion of physiological, environmental, personal data10.1016/j.eswa.2024.123723 ↗
20. Multisensor Data Fusion: A Review of the State-of-the-Art10.1016/j.inffus.2011.08.001 ↗
21. Explainability meets uncertainty quantification: Insights from feature-based model fusion on multimodal time series10.1016/j.inffus.2023.101955 ↗
22. Generalized evidence theory10.1007/s10489-015-0661-2 ↗
23. Classifier Calibration: A survey on how to assess and improve predicted class probabilities10.1007/s10994-023-06336-7 ↗
24. Incorporation of seafarer psychological factors into maritime safety assessment10.1016/j.ocecoaman.2023.106515 ↗
25. Liu, Y., Hou, X., Sourina, O., Konovessis, D., and Krishnan, G. (2016, January 4–6). EEG-based human factors evaluation for maritime simulator-aided assessment. Proceedings of the 3rd International Conference on Maritime Technology and Engineering (MARTECH 2016), Lisbon, Portugal.10.1201/b21890-113 ↗
26. Wang, Z., Zhang, J., Mao, Z., Fan, S., Wang, Z., and Wang, H. (2021, January 22–24). Emotional State Evaluation during Collision Avoidance Operations of Seafarers Using Ship Bridge Simulator and Wearable EEG. Proceedings of the 6th International Conference on Transportation Information and Safety (ICTIS) 2021, Wuhan, China.10.1109/ICTIS54573.2021.9798643 ↗
27. Exploring seafarers’ emotional responses to emergencies: An empirical study using a ship handling simulator10.1016/j.ocecoaman.2023.106736 ↗
28. Lim, W.L., Liu, Y., Subramaniam, S.C.H., Liew, S.H.P., Krishnan, G., Sourina, O., and Wang, L. (2018). EEG-Based Mental Workload and Stress Monitoring of Crew Members in Maritime Virtual Simulator. Transactions on Computational Science XXXII: Special Issue on Cybersecurity and Biometrics, Springer.10.1007/978-3-662-56672-5_2 ↗
29. Deap: A database for emotion analysis; using physiological signals10.1109/T-AFFC.2011.15 ↗
30. Wearable Integrated Online Fusion Learning Filter for Heart PPG Sensing Tracking10.1109/JSEN.2023.3277719 ↗
31. Machine learning-based multimodal fusion recognition of passenger ship seafarers’ workload: A case study of a real navigation experiment10.1016/j.oceaneng.2024.117346 ↗
32. Albuquerque, A., Tiwari, M., Parent, M., Cassani, R., Gagnon, J.-F., Lafond, D., and Falk, T.H. (2020). WAUC: A Multi-Modal Database for Mental Workload Assessment Under Physical Activity. Front. Neurosci., 14.10.3389/fnins.2020.549524 ↗
33. Influence of physiological, psychological and environmental factors on passenger ship seafarer fatigue in real navigation environment10.1016/j.ssci.2023.106293 ↗
34. A review on transfer learning in EEG signal analysis10.1016/j.neucom.2020.09.017 ↗
35. On combining classifiers10.1109/34.667881 ↗
36. Classifiers Combination Techniques: A Comprehensive Review10.1109/ACCESS.2018.2813079 ↗
37. Toufiq, R., and Islam, M.R. (2016, January 8–10). Face recognition system using soft-output classifier fusion method. Proceedings of the 2016 2nd International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE), Rajshahi, Bangladesh.10.1109/ICECTE.2016.7879582 ↗
38. Learn++NC: Combining Ensemble of Classifiers With Dynamically Weighted Consult-and-Vote for Efficient Incremental Learning of New Classes10.1109/TNN.2008.2008326 ↗
39. Classifier combination and score level fusion: Concepts and practical aspects10.1080/19479830903561944 ↗
40. Han, D.Q., Han, C.Z., and Yang, Y. (2007, January 2–4). Combination of heterogeneous multiple classifiers based on evidence theory. Proceedings of the 2007 International Conference on Wavelet Analysis and Pattern Recognition (ICWAPR), Beijing, China.
41. A Novel Ensemble Learning Algorithm Based on D-S Evidence Theory for IoT Security
42. Analysis and prediction of rockburst intensity using improved D-S evidence theory based on multiple machine learning algorithms10.1016/j.tust.2023.105331 ↗
43. Type-2 fuzzy blended improved D-S evidence theory based decision fusion for face recognition10.1016/j.asoc.2022.109179 ↗
44. A machine learning based personalized system for driving state recognition10.1016/j.trc.2019.05.042 ↗
45. Exploration of Machine Learning Methods for Maritime Risk Predictions10.1080/03088839.2023.2209788 ↗
46. A Deep Learning Approach for Port Congestion Estimation and Prediction10.1080/03088839.2022.2057608 ↗
47. A driver stress detection model via data augmentation based on deep convolutional recurrent neural network10.1016/j.eswa.2023.122056 ↗
48. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017;, January 6–11). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning (ICML), Sydney, Australia.
49. A new distance between two bodies of evidence10.1016/S1566-2535(01)00026-4 ↗
50. Selection of optimal EEG electrodes for human emotion recognition10.1016/j.ifacol.2020.12.2753 ↗
51. Dynamic confidence-aware multi-modal emotion recognition10.1109/TAFFC.2023.3340924 ↗
52. Characteristics of real-world ship energy consumption and emissions based on onboard testing10.1016/j.marpolbul.2023.115411 ↗
53. Cogemonet: A cognitive-feature-augmented driver emotion recognition model for smart cockpit10.1109/TCSS.2021.3127935 ↗
54. He, H., Bai, Y., Garcia, E.A., and Li, S. (2008, January 1–8). ADASYN: Adaptive synthetic sampling approach for imbalanced learning. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IJCNN), Hong Kong, China.
55. Frenkel, L., and Goldberger, J. (September, January 29). Network calibration by temperature scaling based on the predicted confidence. Proceedings of the 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia.10.23919/EUSIPCO55093.2022.9909866 ↗
56. An aggressive driving state recognition model using EEG based on stacking ensemble learning
57. Wang, J., Ma, H., and Yan, X. (2023). Rockburst Intensity Classification Prediction Based on Multi-Model Ensemble Learning Algorithms. Mathematics, 11.10.3390/math11040838 ↗
58. Strength of Stacking Technique of Ensemble Learning in Rockburst Prediction with Imbalanced Data: Comparison of Eight Single and Ensemble Models10.1007/s11053-020-09787-0 ↗