Thermal Infrared Imaging-Based Affective Computing and Its Application to Facilitate Human Robot Interaction: A Review
Source abstract
Abstract
Over recent years, robots are increasingly being employed in several aspects of modern society. Among others, social robots have the potential to benefit education, healthcare, and tourism. To achieve this purpose, robots should be able to engage humans, recognize users’ emotions, and to some extent properly react and "behave" in a natural interaction. Most robotics applications primarily use visual information for emotion recognition, which is often based on facial expressions. However, the display of emotional states through facial expression is inherently a voluntary controlled process that is typical of human–human interaction. In fact, humans have not yet learned to use this channel when communicating with a robotic technology. Hence, there is an urgent need to exploit emotion information channels not directly controlled by humans, such as those that can be ascribed to physiological modulations. Thermal infrared imaging-based affective computing has the potential to be the solution to such an issue. It is a validated technology that allows the non-obtrusive monitoring of physiological parameters and from which it might be possible to infer affective states. This review is aimed to outline the advantages and the current research challenges of thermal imaging-based affective computing for human–robot interaction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · EEG and Brain-Computer Interfaces · Non-Invasive Vital Sign Monitoring
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Neuroscience and Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti-Pescara, Via Luigi Polacchi 13, 66100 Chieti, Italy
Next2U s.r.l., Via dei Peligni 137, 65127 Pescara, Italy
Funding
PON SI-ROBOTICS · PON ADAS+
References (100+)
Show source-provided references
1. Sung, J.-Y., Guo, L., Grinter, R.E., and Christensen, H.I. (2007, January 16–19). “My Roomba is Rambo”: Intimate home appliances. Proceedings of the International Conference on Ubiquitous Computing, Innsbruck, Austria.
2. Methodology & themes of human-robot interaction: A growing research field10.5772/5702 ↗
3. Going into the wild in child–robot interaction studies: Issues in social robotic development10.1007/s11370-007-0009-9 ↗
4. Horvitz, E., and Paek, T. (2001, January 13–17). Harnessing models of users’ goals to mediate clarification dialog in spoken language systems. Proceedings of the International Conference on User Modeling, Sonthofen, Germany.10.1007/3-540-44566-8_1 ↗
5. What is a Human? Toward psychological benchmarks in the field of human–robot interaction10.1075/is.8.3.04kah ↗
6. Anthropomorphism and the social robot10.1016/S0921-8890(02)00374-3 ↗
7. Turkle, S. (2010). In good company? On the threshold of robotic companions. Close Engagements with Artificial Companions, John Benjamins.10.1075/nlp.8.03tur ↗
8. Thomasen, K. (2016). Examining the constitutionality of robot-enhanced interrogation. Robot Law, Edward Elgar Publishing.10.4337/9781783476732.00021 ↗
9. Darling, K. (2016). Extending legal protection to social robots: The effects of anthropomorphism, empathy, and violent behavior towards robotic objects. Robot Law, Edward Elgar Publishing.10.4337/9781783476732.00017 ↗
10. Darling, K. (2015). ’Who’s Johnny?’Anthropomorphic Framing in Human-Robot Interaction, Integration, and Policy. Anthr. Fram. Hum. Robot Interact. Integr. Policy March 23 2015 ROBOT ETHICS, 2.10.2139/ssrn.2588669 ↗
11. Press, T.M. (2019, December 27). Affective Computing. Available online: https://mitpress.mit.edu/books/affective-computing.
12. Facial and vocal expressions of emotion10.1146/annurev.psych.54.101601.145102 ↗
13. Fernandes, A., Helawar, R., Lokesh, R., Tari, T., and Shahapurkar, A.V. (2014, January 18–19). Determination of stress using blood pressure and galvanic skin response. Proceedings of the 2014 International Conference on Communication and Network Technologies, 2014, Sivakasi, India.10.1109/CNT.2014.7062747 ↗
14. Measuring emotion: Behavior, feeling, and physiology
15. Cognitive, social, and physiological determinants of emotional state10.1037/h0046234 ↗
16. Knapp, R.B., Kim, J., and André, E. (2011). Physiological signals and their use in augmenting emotion recognition for human–machine interaction. Emotion-Oriented Systems, Springer.10.1007/978-3-642-15184-2_9 ↗
17. Reynolds, C., and Picard, R. (2004, January 24–29). Affective sensors, privacy, and ethical contracts. Proceedings of the CHI’04 Extended Abstracts on Human Factors in Computing Systems, 2004, Vienna, Austria.10.1145/985921.985999 ↗
18. Sebe, N., Sun, Y., Bakker, E., Lew, M.S., Cohen, I., and Huang, T.S. (2004, January 10–13). Towards authentic emotion recognition. Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), Hague, The Netherlands.
19. Human facial expressions as adaptations: Evolutionary questions in facial expression research10.1002/ajpa.20001 ↗
20. Facial displays are tools for social influence10.1016/j.tics.2018.02.006 ↗
21. Wioleta, S. (2013, January 6–8). Using physiological signals for emotion recognition. Proceedings of the 2013 6th International Conference on Human System Interactions (HSI), 2013, Sopot, Poland.10.1109/HSI.2013.6577880 ↗
22. Jerritta, S., Murugappan, M., Nagarajan, R., and Wan, K. (2011, January 4–6). Physiological signals based human emotion recognition: A review. Proceedings of the 2011 IEEE 7th International Colloquium on Signal Processing and its Applications, 2011, Penang, Malaysia.10.1109/CSPA.2011.5759912 ↗
23. Basu, A., Routray, A., Shit, S., and Deb, A.K. (2015, January 17–20). Human emotion recognition from facial thermal image based on fused statistical feature and multi-class SVM. Proceedings of the 2015 Annual IEEE India Conference (INDICON), 2015, New Delhi, India.10.1109/INDICON.2015.7443712 ↗
24. Puri, C., Olson, L., Pavlidis, I., Levine, J., and Starren, J. (2005, January 2–7). StressCam: Non-contact measurement of users’ emotional states through thermal imaging. Proceedings of the CHI’05 extended abstracts on Human factors in computing systems, 2005, Portland, OR, USA.10.1145/1056808.1057007 ↗
25. Emotion recognition from thermal infrared images using deep Boltzmann machine10.1007/s11704-014-3295-3 ↗
26. Thermal imaging to detect physiological indicators of stress in humans
27. Cardone, D., Pinti, P., and Merla, A. (2015). Thermal infrared imaging-based computational psychophysiology for psychometrics. Comput. Math. Methods Med., 2015.10.1155/2015/984353 ↗
28. Engert, V., Merla, A., Grant, J.A., Cardone, D., Tusche, A., and Singer, T. (2014). Exploring the Use of Thermal Infrared Imaging in Human Stress Research. PLoS ONE, 9.10.1371/journal.pone.0090782 ↗
29. Mother and child in synchrony: Thermal facial imprints of autonomic contagion10.1016/j.biopsycho.2011.09.018 ↗
30. “The face of ostracism”: The impact of the social categorization on the thermal facial responses of the target and the observer10.1016/j.actpsy.2015.11.001 ↗
31. Thermal signature of fear conditioning in mild post traumatic stress disorder10.1016/j.neuroscience.2014.02.009 ↗
32. Thermal signatures of voluntary deception in ecological conditions10.1038/srep35174 ↗
33. Aureli, T., Grazia, A., Cardone, D., and Merla, A. (2015). Behavioral and facial thermal variations in 3-to 4-month-old infants during the Still-Face Paradigm. Front. Psychol., 6.10.3389/fpsyg.2015.01586 ↗
34. Perpetuini, D., Cardone, D., Filippini, C., Chiarelli, A.M., and Merla, A. (2019). Modelling Impulse Response Function of Functional Infrared Imaging for General Linear Model Analysis of Autonomic Activity. Sensors, 19.10.3390/s19040849 ↗
35. Perpetuini, D., Cardone, D., Chiarelli, A.M., Filippini, C., Croce, P., Zappasodi, F., Rotunno, L., Anzoletti, N., Zito, M., and Merla, A. (2019). Autonomic impairment in Alzheimer’s disease is revealed by complexity analysis of functional thermal imaging signals during cognitive tasks. Physiol. Meas., 40.10.1088/1361-6579/ab057d ↗
36. Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele University.
37. Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., and Tanaka, F. (2018). Social robots for education: A review. Sci. Robot., 3.10.1126/scirobotics.aat5954 ↗
38. Girotto, V., Lozano, C., Muldner, K., Burleson, W., and Walker, E. (2016, January 7–12). Lessons Learned from In-School Use of rTAG: A Robo-Tangible Learning Environment. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA. Association for Computing Machinery, New York USA.10.1145/2858036.2858454 ↗
39. Movellan, J., Eckhardt, M., Virnes, M., and Rodriguez, A. (2009, January 9–13). Sociable robot improves toddler vocabulary skills. Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, La Jolla, CA, USA.10.1145/1514095.1514189 ↗
40. Tanaka, F., Isshiki, K., Takahashi, F., Uekusa, M., Sei, R., and Hayashi, K. (2015, January 3–5). Pepper learns together with children: Development of an educational application. Proceedings of the 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), Seoul, Korea.10.1109/HUMANOIDS.2015.7363546 ↗
41. Animal-assisted therapy and loneliness in nursing homes: Use of robotic versus living dogs10.1016/j.jamda.2007.11.007 ↗
42. Hodges, H., Fealko, C., and Soares, N. (2020). Autism spectrum disorder: Definition, epidemiology, causes, and clinical evaluation. Transl. Pediatr., 9.10.21037/tp.2019.09.09 ↗
43. State of The Art: A Study of Human-Robot Interaction in Healthcare
44. Why robots? A survey on the roles and benefits of social robots in the therapy of children with autism10.1007/s12369-013-0202-2 ↗
45. Improving therapeutic outcomes in autism spectrum disorders: Enhancing social communication and sensory processing through the use of interactive robots10.1016/j.jpsychires.2017.02.004 ↗
46. Wilson, K.P., and Landa, R.J. (2019). Barriers to Educator Implementation of a Classroom-Based Intervention for Preschoolers with Autism Spectrum Disorder. Front. Educ., 4.10.3389/feduc.2019.00027 ↗
47. Stangl, B., and Pesonen, J. (2018, January 24–26). Consumer Evaluation of Hotel Service Robots. Proceedings of the Information and Communication Technologies in Tourism, Jönköping, Sweden.
48. Awan, I., Younas, M., Franch, X., and Quer, C. (2014, January 27–29). SARA: Singapore’s Automated Responsive Assistant, A Multimodal Dialogue System for Touristic Information. Proceedings of the Mobile Web Information Systems, Barcelona, Spain.
49. Lee, G.G., Kim, H.K., Jeong, M., and Kim, J.-H. (2015). CLARA: A Multifunctional Virtual Agent for Conference Support and Touristic Information. Natural Language Dialog Systems and Intelligent Assistants, Springer International Publishing.
50. de Kervenoael, R., Hasan, R., Schwob, A., and Goh, E. (2020). Leveraging human-robot interaction in hospitality services: Incorporating the role of perceived value, empathy, and information sharing into visitors’ intentions to use social robots. Tour. Manag., 78.10.1016/j.tourman.2019.104042 ↗
51. Cross, E.S., Hortensius, R., and Wykowska, A. (2019). From social brains to social robots: Applying neurocognitive insights to human–robot interaction. Philos. Trans. R. Soc. B Biol. Sci., 374.10.1098/rstb.2018.0024 ↗
52. Cognitive heat: Exploring the usage of thermal imaging to unobtrusively estimate cognitive load10.1145/3130898 ↗
53. The mental and subjective skin: Emotion, empathy, feelings and thermography10.1016/j.concog.2015.04.003 ↗
54. Pavlidis, I., Tsiamyrtzis, P., Shastri, D., Wesley, A., Zhou, Y., Lindner, P., Buddharaju, P., Joseph, R., Mandapati, A., and Dunkin, B. (2012). Fast by nature-how stress patterns define human experience and performance in dexterous tasks. Sci. Rep., 2.10.1038/srep00305 ↗
55. Thermal infrared imaging in psychophysiology: Potentialities and limits10.1111/psyp.12243 ↗
56. Human emotions detection based on a smart-thermal system of thermographic images10.1016/j.infrared.2017.01.002 ↗
57. Goulart, C., Valadão, C., Delisle-Rodriguez, D., Caldeira, E., and Bastos, T. (2019). Emotion analysis in children through facial emissivity of infrared thermal imaging. PLoS ONE, 14.10.1371/journal.pone.0212928 ↗
58. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament10.1007/BF02686918 ↗
59. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology10.1017/S0954579405050340 ↗
60. Marcus, A., and Wang, W. (2019, January 26–31). A Study of Emotional Communication of Emoticon Based on Russell’s Circumplex Model of Affect. Proceedings of the Design, User Experience, and Usability Design Philosophy and Theory, Orlando, FL, US.
61. Landowska, A. (2018). Towards new mappings between emotion representation models. Appl. Sci., 8.10.3390/app8020274 ↗
62. Emotional category data on images from the International Affective Picture System10.3758/BF03192732 ↗
63. Kosonogov, V., Zorzi, L.D., Honoré, J., Martínez-Velázquez, E.S., Nandrino, J.-L., Martinez-Selva, J.M., and Sequeira, H. (2017). Facial thermal variations: A new marker of emotional arousal. PLoS ONE, 12.10.1371/journal.pone.0183592 ↗
64. Diaz-Piedra, C., Gomez-Milan, E., and Di Stasi, L.L. (2019). Nasal skin temperature reveals changes in arousal levels due to time on task: An experimental thermal infrared imaging study. Appl. Ergon., 81.10.1016/j.apergo.2019.06.001 ↗
65. Evaluation of dynamics of forehead skin temperature under induced drowsiness10.1002/tee.22423 ↗
66. Stress in interactive applications: Analysis of the valence-arousal space based on physiological signals and self-reported data10.1007/s11042-016-3637-2 ↗
67. Ioannou, S., Ebisch, S., Aureli, T., Bafunno, D., Ioannides, H.A., Cardone, D., Manini, B., Romani, G.L., Gallese, V., and Merla, A. (2013). The Autonomic Signature of Guilt in Children: A Thermal Infrared Imaging Study. PLoS ONE, 8.10.1371/journal.pone.0079440 ↗
68. Manini, B., Cardone, D., Ebisch, S., Bafunno, D., Aureli, T., and Merla, A. (2013). Mom feels what her child feels: Thermal signatures of vicarious autonomic response while watching children in a stressful situation. Front. Hum. Neurosci., 7.10.3389/fnhum.2013.00299 ↗
69. Contact-Free Measurement of Cardiac Pulse Based on the Analysis of Thermal Imagery10.1109/TBME.2007.891930 ↗
70. A novel method for extracting respiration rate and relative tidal volume from infrared thermography10.1111/j.1469-8986.2010.01167.x ↗
71. Noncontact measurement of breathing function10.1109/MEMB.2006.1636352 ↗
72. Remote monitoring of breathing dynamics using infrared thermography10.1364/BOE.6.004378 ↗
73. Thermistor at a distance: Unobtrusive measurement of breathing
74. Neonatal non-contact respiratory monitoring based on real-time infrared thermography10.1186/1475-925X-10-93 ↗
75. Monitoring the Variation in Driver Respiration Rate from Wakefulness to Drowsiness: A Non-Intrusive Method for Drowsiness Detection Using Thermal Imaging
76. Cho, Y., and Bianchi-Berthouze, N. (2019). Physiological and Affective Computing through Thermal Imaging: A Survey. ArXiv, 1908.
77. Thermography Guidelines (2020, February 18). Standards and Protocols. Available online: http://www.iact-org.org/professionals/thermog-guidelines.html.
78. Cho, Y., Julier, S.J., and Bianchi-Berthouze, N. (2019). Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging. JMIR Ment. Health, 6.10.2196/10140 ↗
79. Cho, Y. (2017, January 23–26). Automated mental stress recognition through mobile thermal imaging. Proceedings of the 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, USA.10.1109/ACII.2017.8273662 ↗
80. Cho, Y., Bianchi-Berthouze, N., and Julier, S.J. (2017, January 23–26). DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. Proceedings of the 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, USA.10.1109/ACII.2017.8273639 ↗
81. A Portable Personality Recognizer Based on Affective State Classification Using Spectral Fusion of Features10.1109/TAFFC.2018.2828845 ↗
82. Vinciguerra, V., Ambra, E., Maddiona, L., Romeo, M., Mazzillo, M., Rundo, F., Fallica, G., di Pompeo, F., Chiarelli, A.M., and Zappasodi, F. (2018, January 21–23). PPG/ECG multisite combo system based on SiPM technology. Proceedings of the Convegno Nazionale Sensori, Catania, Italy.10.1007/978-3-030-04324-7_44 ↗
83. Data-driven assessment of cardiovascular ageing through multisite photoplethysmography and electrocardiography10.1016/j.medengphy.2019.07.009 ↗
84. Ruminski, J., and Kwasniewska, A. (2017). Evaluation of respiration rate using thermal imaging in mobile conditions. Application of Infrared to Biomedical Sciences, Springer.10.1007/978-981-10-3147-2_18 ↗
85. Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging10.1364/BOE.8.004480 ↗
86. Goulart, C., Valadão, C., Delisle-Rodriguez, D., Funayama, D., Favarato, A., Baldo, G., Binotte, V., Caldeira, E., and Bastos-Filho, T. (2019). Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction. Sensors, 19.10.3390/s19132844 ↗
87. Filippini, C., Spadolini, E., Cardone, D., and Merla, A. (2019, January 16–19). Thermal Imaging Based Affective Computing for Educational Robot. Proceedings of the Multidisciplinary Digital Publishing Institute Proceedings, Florence, Italy.10.3390/proceedings2019027027 ↗
88. Merla, A. (2014). Thermal expression of intersubjectivity offers new possibilities to human–machine and technologically mediated interactions. Front. Psychol., 5.10.3389/fpsyg.2014.00802 ↗
89. Sorostinean, M., Ferland, F., and Tapus, A. (2015, January 3–5). Reliable stress measurement using face temperature variation with a thermal camera in human-robot interaction. Proceedings of the 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), Seoul, Korea.10.1109/HUMANOIDS.2015.7363516 ↗
90. Agrigoroaie, R., and Tapus, A. (2017, January 6–9). Detecting Deception in a Human-Robot Interaction Scenario. Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria.10.1145/3029798.3038376 ↗
91. Boccanfuso, L., Wang, Q., Leite, I., Li, B., Torres, C., Chen, L., Salomons, N., Foster, C., Barney, E., and Ahn, Y.A. (2016, January 26–31). A thermal emotion classifier for improved human-robot interaction. Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, USA.10.1109/ROMAN.2016.7745198 ↗
92. The mental nose and the Pinocchio effect: Thermography, planning, anxiety, and lies10.1002/jip.1505 ↗
93. The Pinocchio effect and the Cold Stress Test: Lies and thermography10.1111/psyp.12956 ↗
94. Merla, A., and Romani, G.L. (2007, January 22–26). Thermal Signatures of Emotional Arousal: A Functional Infrared Imaging Study. Proceedings of the 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Lyon, France.10.1109/IEMBS.2007.4352270 ↗
95. The Polyvagal Theory: Phylogenetic contributions to social behavior10.1016/S0031-9384(03)00156-2 ↗
96. Scassellati, B., Brawer, J., Tsui, K., Nasihati Gilani, S., Malzkuhn, M., Manini, B., Stone, A., Kartheiser, G., Merla, A., and Shapiro, A. (2018, January 21–26). Teaching Language to Deaf Infants with a Robot and a Virtual Human. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal QC, Canada.10.1145/3173574.3174127 ↗
97. The “Perceptual Wedge Hypothesis” as the basis for bilingual babies’ phonetic processing advantage: New insights from fNIRS brain imaging10.1016/j.bandl.2011.05.003 ↗
98. Nasihati Gilani, S., Traum, D., Merla, A., Hee, E., Walker, Z., Manini, B., Gallagher, G., and Petitto, L.-A. (2018, January 16–20). Multimodal Dialogue Management for Multiparty Interaction with Infants. Proceedings of the 20th ACM International Conference on Multimodal Interaction, Boulder, CO, USA.10.1145/3242969.3243029 ↗
99. Nasihati Gilani, S., Traum, D., Sortino, R., Gallagher, G., Aaron-lozano, K., Padilla, C., Shapiro, A., Lamberton, J., and Petitto, L. (2019, January 13–17). Can a Virtual Human Facilitate Language Learning in a Young Baby?. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, Montreal, QC, Canada.
100. Nasihati Gilani, S., Traum, D., Sortino, R., Gallagher, G., Aaron-Lozano, K., Padilla, C., Shapiro, A., Lamberton, J., and Petitto, L.-A. (2019, January 2’5). Can a Signing Virtual Human Engage a Baby’s Attention?. Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, Paris, France.10.1145/3308532.3329463 ↗