journal article·2020-08-15Open access

Driver Stress State Evaluation by Means of Thermal Imaging: A Supervised Machine Learning Approach Based on ECG Signal

Daniela Cardone, David Perpetuini, Chiara Filippini, Edoardo Spadolini, Lorenza Mancini, Antonio Maria Chiarelli, Arcangelo Merla

Applied Sciences· Volume 10· Issue 16

Abstract

Traffic accidents determine a large number of injuries, sometimes fatal, every year. Among other factors affecting a driver’s performance, an important role is played by stress which can decrease decision-making capabilities and situational awareness. In this perspective, it would be beneficial to develop a non-invasive driver stress monitoring system able to recognize the driver’s altered state. In this study, a contactless procedure for drivers’ stress state assessment by means of thermal infrared imaging was investigated. Thermal imaging was acquired during an experiment on a driving simulator, and thermal features of stress were investigated with comparison to a gold-standard metric (i.e., the stress index, SI) extracted from contact electrocardiography (ECG). A data-driven multivariate machine learning approach based on a non-linear support vector regression (SVR) was employed to estimate the SI through thermal features extracted from facial regions of interest (i.e., nose tip, nostrils, glabella). The predicted SI showed a good correlation with the real SI (r = 0.61, p = ~0). A two-level classification of the stress state (STRESS, SI ≥ 150, versus NO STRESS, SI < 150) was then performed based on the predicted SI. The ROC analysis showed a good classification performance with an AUC of 0.80, a sensitivity of 77%, and a specificity of 78%.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Infrared Thermography in Medicine · Heart Rate Variability and Autonomic Control · Sleep and Work-Related Fatigue

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Neurosciences, Imaging and Clinical Sciences (DNISC), University G. d’Annunzio of Chieti-Pescara, 66100 Chieti, Italy

Next2U s.r.l., 65127 Pescara, Italy

Funding

Ministero dell’Istruzione, dell’Università e della Ricerca · Horizon 2020

References (60)

Show source-provided references
1. WHO (2018). Global Status Report on Road Safety 2018: Summary (No. WHO/NMH/NVI/18.20), WHO.
2. Three decades of driver assistance systems: Review and future perspectives10.1109/MITS.2014.2336271 ↗
3. Object Recognition based interpolation with 3d lidar and vision for autonomous driving of an intelligent vehicle10.1109/ACCESS.2020.2982681 ↗
4. Catten, J.C., and McClellan, S. (2012). System and Method for Alerting Drivers to Road Conditions. (8,188,887), U.S. Patent.
5. Driver-vehicle interfaces and interaction: Where are they going?10.1007/s12544-009-0009-2 ↗
6. Traffic behavior recognition using the pachinko allocation model10.3390/s150716040 ↗
7. Urzaiz, G., Ochoa, S.F., Bravo, J., Chen, L.L., and Oliveira, J. (2013). Risky driving detection through urban mobility traces: A preliminary approach. Context-Awareness and Context-Driven Interaction, Proceedings of the Ubiquitous Computing and Ambient Intelligence, Carrillo, CR, USA, 2–6 December 2013, Springer.
8. (2020, August 14). Distracted Driving, Available online: https://www.nhtsa.gov/risky-driving/distracted-driving.
9. Driver crash risk factors and prevalence evaluation using naturalistic driving data10.1073/pnas.1513271113 ↗
10. Guettas, A., Ayad, S., and Kazar, O. (2019, January 23–24). Driver state monitoring system: A review. Proceedings of the 4th International Conference on Big Data and Internet of Things, Tangier, Morocco.10.1145/3372938.3372966 ↗
11. Happy-anger emotions classifications from electrocardiogram signal for automobile driving safety and awareness10.1016/j.jth.2017.11.001 ↗
12. Wearable mobile-based emotional response-monitoring system for drivers10.1109/THMS.2017.2658442 ↗
13. Automatic driver sleepiness detection using EEG, EOG and contextual information10.1016/j.eswa.2018.07.054 ↗
14. EEG classification of driver mental states by deep learning10.1007/s11571-018-9496-y ↗
15. Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers10.1016/j.eswa.2017.01.040 ↗
16. Detecting stress during real-world driving tasks using physiological sensors10.1109/TITS.2005.848368 ↗
17. Munla, N., Khalil, M., Shahin, A., and Mourad, A. (2015, January 16–18). Driver stress level detection using HRV analysis. Proceedings of the 2015 International Conference on Advances in Biomedical Engineering (ICABME), Beirut, Lebanon.10.1109/ICABME.2015.7323251 ↗
18. Baevsky, R.M., and Chernikova, A.G. (2017). Heart rate variability analysis: Physiological foundations and main methods. Cardiometry, 66–76.10.12710/cardiometry.2017.10.6676 ↗
19. Cardone, D., and Merla, A. (2017). New frontiers for applications of thermal infrared imaging devices: Computational psychopshysiology in the neurosciences. Sensors, 17.10.3390/s17051042 ↗
20. Filippini, C., Perpetuini, D., Cardone, D., Chiarelli, A.M., and Merla, A. (2020). Thermal infrared imaging-based affective computing and its application to facilitate human robot interaction: A review. Appl. Sci., 10.10.3390/app10082924 ↗
21. Engert, V., Merla, A., Grant, J.A., Cardone, D., Tusche, A., and Singer, T. (2014). Exploring the use of thermal infrared imaging in human stress research. PLoS ONE, 9.10.1371/journal.pone.0090782 ↗
22. Human emotions detection based on a smart-thermal system of thermographic images10.1016/j.infrared.2017.01.002 ↗
23. Perpetuini, D., Cardone, D., Bucco, R., Zito, M., and Merla, A. (2020, June 25). Assessment of the Autonomic Response in Alzheimer’s Patients During the Execution of Memory Tasks: A Functional Thermal Imaging Study. Available online: https://www.ingentaconnect.com/content/ben/car/2018/00000015/00000010/art00007.
24. Puri, C., Olson, L., Pavlidis, I., Levine, J., and Starren, J. (2005). StressCam: Non-contact measurement of users’ emotional states through thermal imaging. CHI’05 Extended Abstracts on Human Factors in Computing Systems, ACM.10.1145/1056808.1057007 ↗
25. Fast by nature-how stress patterns define human experience and performance in dexterous tasks10.1038/srep00305 ↗
26. Kang, J., McGinley, J.A., McFadyen, G., and Babski-Reeves, K. (2006, January 27–30). Determining learning level and effective training times using thermography. Proceedings of the Army Science Conference, Orlando, FL, USA.
27. Stemberger, J., Allison, R.S., and Schnell, T. (June, January 31). Thermal imaging as a way to classify cognitive workload. Proceedings of the 2010 Canadian Conference on Computer and Robot Vision, Ottawa, ON, Canada.10.1109/CRV.2010.37 ↗
28. Monitoring the variation in driver respiration rate from wakefulness to drowsiness: A non-intrusive method for drowsiness detection using thermal imaging
29. Driver’s fatigue recognition based on yawn detection in thermal images10.1016/j.neucom.2019.02.014 ↗
30. Discriminating drivers’ emotions through the dimension of power: Evidence from facial infrared thermography and peripheral physiological measurements10.1016/j.trf.2019.04.003 ↗
31. Yamakoshi, T., Yamakoshi, K., Tanaka, S., Nogawa, M., Park, S.B., Shibata, M., Sawada, Y., Rolfe, P., and Hirose, Y. (2008, January 20–24). Feasibility study on driver’s stress detection from differential skin temperature measurement. Proceedings of the 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vancouver, BC, Canada.10.1109/IEMBS.2008.4649346 ↗
32. Dissecting driver behaviors under cognitive, emotional, sensorimotor, and mixed stressors10.1038/srep25651 ↗
33. A literature review on supervised machine learning algorithms and boosting process
34. (2000). World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects. JAMA, 284, 3043–3045.10.1001/jama.284.23.3043 ↗
35. (2020, June 26). City Car Driving—Car Driving Simulator, PC Game. Available online: https://citycardriving.com/.
36. Conover, M.B. (2002). Understanding Electrocardiography, Elsevier Health Sciences.
37. Vander Sloten, J., Verdonck, P., Nyssen, M., and Haueisen, J. (2008, January 23–27). Kubios HRV—A Software for Advanced Heart Rate Variability Analysis. Proceedings of the 4th European Conference of the International Federation for Medical and Biological Engineering, Antwerp, Belgium.10.1007/978-3-540-89208-3 ↗
38. Bradski, G., and Kaehler, A. (2008). Learning OpenCV: Computer Vision with the OpenCV Library, O’Reilly Media, Inc.
39. Filippini, C., Spadolini, E., Cardone, D., Bianchi, D., Preziuso, M., Sciarretta, C., del Cimmuto, V., Lisciani, D., and Merla, A. (2020). Facilitating the child–robot interaction by endowing the robot with the capability of understanding the child engagement: The case of mio amico robot. Int. J. Soc. Robot., 1–13.10.1007/s12369-020-00661-w ↗
40. Baltrušaitis, T., Robinson, P., and Morency, L.-P. (2016, January 7–9). OpenFace: An open source facial behavior analysis toolkit. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.10.1109/WACV.2016.7477553 ↗
41. Openface: A general-purpose face recognition library with mobile applications
42. Baltrusaitis, T., Robinson, P., and Morency, L.-P. (2013, January 2–8). Constrained Local Neural Fields for Robust Facial Landmark Detection in the Wild. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, Australia.10.1109/ICCVW.2013.54 ↗
43. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks10.1109/LSP.2016.2603342 ↗
44. Thermal infrared imaging in psychophysiology: Potentialities and limits10.1111/psyp.12243 ↗
45. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC10.1007/s11222-016-9696-4 ↗
46. Crone, S.F., Guajardo, J., and Weber, R. (2006, January 21–24). A study on the ability of support vector regression and neural networks to forecast basic time series patterns. Proceedings of the IFIP International Conference on Artificial Intelligence in Theory and Practice, Santiago, Chile.
47. Bootstrap re-sampling for unbalanced data in supervised learning10.1016/S0377-2217(00)00244-7 ↗
48. Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine10.1093/clinchem/39.4.561 ↗
49. Baltrušaitis, T. (2014). Automatic Facial Expression Analysis. [Ph.D. Thesis, University of Cambridge].
50. The correlation between students’ interest and learning outcomes in biology
51. Widanti, N., Sumanto, B., Rosa, P., and Fathur Miftahudin, M. (2015, January 28–30). Stress level detection using heart rate, blood pressure, and GSR and stress therapy by utilizing infrared. Proceedings of the 2015 International Conference on Industrial Instrumentation and Control (ICIC), Pune, India.10.1109/IIC.2015.7150752 ↗
52. Coronary vasoconstriction induced by mental stress (simulated public speaking)10.1016/S0002-9149(99)80590-6 ↗
53. Statistical notes for clinical researchers: Assessing normal distribution (2) using skewness and kurtosis10.5395/rde.2013.38.1.52 ↗
54. Remote monitoring of breathing dynamics using infrared thermography10.1364/BOE.6.004378 ↗
55. Widjaja, D., Orini, M., Vlemincx, E., and Van Huffel, S. (2013). Cardiorespiratory Dynamic Response to Mental Stress: A Multivariate Time-Frequency Analysis. Comput. Math. Methods Med., 2013.10.1155/2013/451857 ↗
56. Sigh rate and respiratory variability during mental load and sustained attention10.1111/j.1469-8986.2010.01043.x ↗
57. Lohani, M., Payne, B.R., and Strayer, D.L. (2019). A review of psychophysiological measures to assess cognitive states in real-world driving. Front. Hum. Neurosci., 13.10.3389/fnhum.2019.00057 ↗
58. Driver stress as influenced by driving maneuvers and roadway conditions10.1016/j.trf.2006.09.002 ↗
59. Age and gender differences in stress responses during simulated driving10.1177/154193129904301802 ↗
60. Driving performance in cold, warm, and thermoneutral environments10.1016/S0003-6870(03)00055-3 ↗