Test–Retest Reliability in Automated Emotional Facial Expression Analysis: Exploring FaceReader 8.0 on Data from Typically Developing Children and Children with Autism
Source abstract
Abstract
Automated emotional facial expression analysis (AEFEA) is used widely in applied research, including the development of screening/diagnostic systems for atypical human neurodevelopmental conditions. The validity of AEFEA systems has been systematically studied, but their test–retest reliability has not been researched thus far. We explored the test–retest reliability of a specific AEFEA software, Noldus FaceReader 8.0 (FR8; by Noldus Information Technology). We collected intensity estimates for 8 repeated emotions through FR8 from facial video recordings of 60 children: 31 typically developing children and 29 children with autism spectrum disorder. Test–retest reliability was imperfect in 20% of cases, affecting a substantial proportion of data points; however, the test–retest differences were small. This shows that the test–retest reliability of FR8 is high but not perfect. A proportion of cases which initially failed to show perfect test–retest reliability reached it in a subsequent analysis by FR8. This suggests that repeated analyses by FR8 can, in some cases, lead to the “stabilization” of emotion intensity datasets. Under ANOVA, the test–retest differences did not influence the pattern of cross-emotion and cross-group effects and interactions. Our study does not question the validity of previous results gained by AEFEA technology, but it shows that further exploration of the test–retest reliability of AEFEA systems is desirable.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Autism Spectrum Disorder Research · Emotion and Mood Recognition · Face Recognition and Perception
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Doctoral School of Psychology, ELTE Eötvös Loránd University, 1053 Budapest, Hungary
Institute of Psychology, ELTE Eötvös Loránd University, 1053 Budapest, Hungary
Institute for the Psychology of Special Needs, ELTE Eötvös Loránd University, 1053 Budapest, Hungary
HAS-ELTE ‘Autism in Education’ Research Group, 1053 Budapest, Hungary
Institute of Special Needs Education for People with Atypical Behavior and Cognition, ELTE Eötvös Loránd University, 1053 Budapest, Hungary
Funding
EIT ICT Labs Hungarian Node · European Union · National Research—Development and Innovation Office · ELTE University · IndaGaléria (Budapest) · Hungarian Academy of Sciences · Foundation for the Development of Special Education (GYFA; Budapest)
References (71)
Show source-provided references
1. Coan, J.A., and Allen, J.J.B. (2007). Use of automated facial image analysis for measurement of emotion expression. Handbook of Emotion Elicitation and Assessment, Oxford University Press.10.1093/oso/9780195169157.003.0015 ↗
2. The Promises and Perils of Automated Facial Action Coding in Studying Children’s Emotions10.1037/dev0000728 ↗
3. Facial Expression Analysis with AFFDEX and FACET: A Validation Study10.3758/s13428-017-0996-1 ↗
4. Mandal, M.K., and Awasthi, A. (2015). Automatic Facial Expression Analysis. Understanding Facial Expressions in Communication, Springer.10.1007/978-81-322-1934-7 ↗
5. Automated Facial Coding: Validation of Basic Emotions and FACS AUs in FaceReader10.1037/npe0000028 ↗
6. Assessing the Effectiveness of Automated Emotion Recognition in Adults and Children for Clinical Investigation10.3389/fnhum.2020.00070 ↗
7. Vallverdú, J. (2015). Automated Recognition of Emotion appraisals. Handbook of Research on Synthesizing Human Emotion in Intelligent Systems and Robotics, IGI Global. Advances in Computational Intelligence and Robotics.10.4018/978-1-4666-7278-9 ↗
8. The Facial Action Coding System for Characterization of Human Affective Response to Consumer Product-Based Stimuli: A Systematic Review10.3389/fpsyg.2020.00920 ↗
9. Facial Expression to Emotional Stimuli in Non-Psychotic Disorders: A Systematic Review and Meta-Analysis10.1016/j.neubiorev.2016.02.015 ↗
10. Girard, J.M., Cohn, J.F., Mahoor, M.H., Mavadati, S., and Rosenwald, D.P. (2013, January 22–26). Social Risk and Depression: Evidence from Manual and Automatic Facial Expression Analysis. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.10.1109/FG.2013.6553748 ↗
11. Measuring Facial Movement10.1007/BF01115465 ↗
12. Samadiani, N., Huang, G., Cai, B., Luo, W., Chi, C.-H., Xiang, Y., and He, J. (2019). A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data. Sensors, 19.10.3390/s19081863 ↗
13. Ekman, P., and Rosenberg, E.L. (2005). What the Face Reveals Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press.10.1093/acprof:oso/9780195179644.001.0001 ↗
14. A psychometric evaluation of the facial action coding system for assessing spontaneous expression10.1023/A:1010671109788 ↗
15. Meta-Analysis of the First Facial Expression Recognition Challenge10.1109/TSMCB.2012.2200675 ↗
16. Kisačanin, B., Pavlović, V., and Huang, T.S. (2005). Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures. Real-Time Vision for Human-Computer Interaction, Springer.10.1007/0-387-27890-7 ↗
17. Predicting Ad Liking and Purchase Intent: Large-Scale Analysis of Facial Responses to Ads10.1109/TAFFC.2014.2384198 ↗
18. A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions10.1109/TPAMI.2008.52 ↗
19. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., and Bartlett, M. (2011, January 21–25). The Computer Expression Recognition Toolbox (CERT). Proceedings of the Face and Gesture 2011, Santa Barbara, CA, USA.10.1109/FG.2011.5771414 ↗
20. Den Uyl, M.J., and van Kuilenburg, H. (September, January 30). The FaceReader: Online facial expression recognition. Proceedings of the Measuring Behavior, Wageningen, The Netherlands. Available online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.116.5261&rep=rep1&type=pdf.
21. Dupré, D., Krumhuber, E.G., Küster, D., and McKeown, G.J. (2020). A Performance Comparison of Eight Commercially Available Automatic Classifiers for Facial Affect Recognition. PLoS ONE, 15.10.1371/journal.pone.0231968 ↗
22. Kawulok, M., Celebi, M.E., and Smolka, B. (2016). Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition. Advances in Face Detection and Facial Image Analysis, Springer International Publishing.10.1007/978-3-319-25958-1 ↗
23. Borges, N., Lindblom, L., Clarke, B., Gander, A., and Lowe, R. (2019, January 3–6). Classifying Confusion: Autodetection of Communicative Misunderstandings Using Facial Action Units. Proceedings of the 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), Cambridge, UK.10.1109/ACIIW.2019.8925037 ↗
24. Calvo, R., D’Mello, S., Gratch, J., and Kappas, A. (2015). Automated Face Analysis for Affective Computing. The Oxford Handbook of Affective Computing, Oxford University Press.10.1093/oxfordhb/9780199942237.013.020 ↗
25. A Comparison of the Affectiva IMotions Facial Expression Analysis Software with EMG for Identifying Facial Expressions of Emotion10.3389/fpsyg.2020.00329 ↗
26. Beumer, G.M., Tao, Q., Bazen, A.M., and Veldhuis, R.N.J. (2006, January 10–12). A Landmark Paper in Face Recognition. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR06), Southampton, UK.
27. Active Appearance Models10.1109/34.927467 ↗
28. Dupré, D., Andelic, N., Morrison, G., and McKeown, G. (2018, January 19–23). Accuracy of Three Commercial Automatic Emotion Recognition Systems across Different Individuals and Their Facial Expressions. Proceedings of the 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Athens, Greece.10.1109/PERCOMW.2018.8480127 ↗
29. Viola, P., and Jones, M. (2001, January 8–14). Rapid Object Detection Using a Boosted Cascade of Simple Features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), Kauai, HI, USA.
30. A Model Based Method for Automatic Facial Expression Recognition
31. Lundqvist, D., Flykt, A., and Öhman, A. (1998). The Karolinska Directed Emotional Faces—KDEF (CD ROM), Karolinska Institute, Department of Clinical Neuroscience, Psychology Section.10.1037/t27732-000 ↗
32. Warsaw Set of Emotional Facial Expression Pictures: A Validation Study of Facial Display Photographs10.3389/fpsyg.2014.01516 ↗
33. Moving Faces, Looking Places: Validation of the Amsterdam Dynamic Facial Expression Set (ADFES)10.1037/a0023853 ↗
34. Automated Facial Coding Software Outperforms People in Recognizing Neutral Faces as Neutral from Standardized Datasets10.3389/fpsyg.2015.01386 ↗
35. Skiendziel, T., Rösch, A.G., and Schultheiss, O.C. (2019). Assessing the Convergent Validity between the Automated Emotion Recognition Software Noldus FaceReader 7 and Facial Action Coding System Scoring. PLoS ONE, 14.10.1371/journal.pone.0223905 ↗
36. Rösch, A.G. (2012). The Influence of Implicit Motives on the Expression and Perception of Facial Expressions of Emotion [Der Einfluss impliziter Motive auf den Ausdruck und die Wahrnehmung Emotionaler Gesichtsausdrücke]. [Ph.D. Thesis, Friedrich-Alexander University]. Available online: https://opus4.kobv.de/opus4-fau/frontdoor/index/index/docId/2304.
37. Coan, J.A., and Allen, J.J.B. (2007). Observer-based measurement of facial expression with the facial action coding system. Handbook of Emotion Elicitation and Assessment, Oxford University Press.10.1093/oso/9780195169157.003.0014 ↗
38. Reliability and Validity of Machine Vision for the Assessment of Facial Expressions10.1016/j.cogsys.2019.03.009 ↗
39. Terzis, V., Moridis, C.N., and Economides, A.A. (2010, January 24–27). Measuring Instant Emotions during a Self-Assessment Test: The Use of FaceReader. Proceedings of the 7th International Conference on Methods and Techniques in Behavioral Research—MB ’10, Eindhoven, The Netherlands.10.1145/1931344.1931362 ↗
40. D’Arcey, J.T. (2013). Assessing the Validity of FaceReader Using Facial EMG, California State University. Available online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.916.6560&rep=rep1&type=pdf.
41. Owada, K., Kojima, M., Yassin, W., Kuroda, M., Kawakubo, Y., Kuwabara, H., Kano, Y., and Yamasue, H. (2018). Computer-Analyzed Facial Expression as a Surrogate Marker for Autism Spectrum Social Core Symptoms. PLoS ONE, 13.10.1371/journal.pone.0190442 ↗
42. Digital Behavioral Phenotyping Detects Atypical Pattern of Facial Expression in Toddlers with Autism10.1002/aur.2391 ↗
43. Facial Expression in Patients with Bipolar Disorder and Schizophrenia in Response to Emotional Stimuli: A Partially Shared Cognitive and Social Deficit of the Two Disorders10.2147/NDT.S46525 ↗
44. Leppanen, J., Dapelo, M.M., Davies, H., Lang, K., Treasure, J., and Tchanturia, K. (2017). Computerised Analysis of Facial Emotion Expression in Eating Disorders. PLoS ONE, 12.10.1371/journal.pone.0178972 ↗
45. Craniofacial Morphology of Children with Williams Syndrome10.1597/1545-1569_1993_030_0343_cmocww_2.3.co_2 ↗
46. Digital Facial Dysmorphology for Genetic Screening: Hierarchical Constrained Local Model Using ICA10.1016/j.media.2014.04.002 ↗
47. American Psychiatric Association (2013). Diagnostic and Statistical Manual of Mental Disorders: DSM-5, American Psychiatric Association. [5th ed.].10.1176/appi.books.9780890425596 ↗
48. Facial Phenotypes in Subgroups of Prepubertal Boys with Autism Spectrum Disorders Are Correlated with Clinical Phenotypes10.1186/2040-2392-2-15 ↗
49. Autism Is a Prenatal Disorder: Evidence from Late Gestation Brain Overgrowth: Prenatal Brain Overgrowth in Autism10.1002/aur.2036 ↗
50. Head Circumference and Brain Size in Autism Spectrum Disorder: A Systematic Review and Meta-Analysis10.1016/j.pscychresns.2015.08.016 ↗
51. Hypermasculinised Facial Morphology in Boys and Girls with Autism Spectrum Disorder and Its Association with Symptomatology10.1038/s41598-017-09939-y ↗
52. Tripi, G., Roux, S., Matranga, D., Maniscalco, L., Glorioso, P., Bonnet-Brilhault, F., and Roccella, M. (2019). Cranio-Facial Characteristics in Children with Autism Spectrum Disorders (ASD). JCM, 8.10.3390/jcm8050641 ↗
53. Children with Autism Spectrum Disorder Produce More Ambiguous and Less Socially Meaningful Facial Expressions: An Experimental Study Using Random Forest Classifiers10.1186/s13229-020-0312-2 ↗
54. Emotional Facial and Vocal Expressions During Story Retelling by Children and Adolescents with High-Functioning Autism10.1044/1092-4388(2012/12-0067) ↗
55. Sharing Humour and Laughter in Autism and Down’s Syndrome10.1348/000712602162553 ↗
56. Facial Expression Production in Autism: A Meta-Analysis: Facial Expression Production in Autism10.1002/aur.2037 ↗
57. “Putting on My Best Normal”: Social Camouflaging in Adults with Autism Spectrum Conditions10.1007/s10803-017-3166-5 ↗
58. Can Automated Facial Expression Analysis Show Differences Between Autism and Typical Functioning?
59. Evidence-Based Development and First Usability Testing of a Social Serious Game Based Multi-Modal System for Early Screening for Atypical Socio-Cognitive Development
60. Data Quality as a Bottleneck in Developing a Social-Serious-Game-Based Multi-Modal System for Early Screening for ‘High Functioning’ Cases of Autism Spectrum Condition10.1007/978-3-319-41267-2_51 ↗
61. Automated vs Human Recognition of Emotional Facial Expressions of High-Functioning Children with Autism in a Diagnostic-Technological Context: Explorations via a Bottom-Up Approach10.1007/978-3-319-94277-3_72 ↗
62. Rutter, M., Bailey, A., and Lord, C. (2003). The Social Communication Questionnaire: Manual, Western Psychological Services.
63. Lord, C., Rutter, M., DiLavore, P.C., and Risi, S. (1999). Autism Diagnostic Observation Schedule, Western Psychological Services.10.1037/t17256-000 ↗
64. Le Couteur, A., Lord, C., and Rutter, M. (2003). The Autism Diagnostic Interview-Revised, Western Psychological Services.
65. Roid, G.H., and Miller, L.J. (1997). Leiter international performance scale-revised: Examiner’s manual. Leiter International Performance Scale—Revised, Wood Dale.
66. Bishop, D. (1983). The Test for Reception of Grammar, Medical Research Council, Applied Psychology Unit.
67. TROG-H: Új sztenderdizált módszer a nyelvtani megértés fejlődésének vizsgálatára
68. Dominance Statistics: Ordinal Analyses to Answer Ordinal Questions10.1037/0033-2909.114.3.494 ↗
69. Deception and Sabotage in Autistic, Retarded and Normal Children10.1111/j.1469-7610.1992.tb00893.x ↗
70. Facial expression production and recognition in autism spectrum disorders: A shifting landscape10.1016/j.chc.2020.02.006 ↗
71. Automatic Emotion and Attention Analysis of Young Children at Home: A ResearchKit Autism Feasibility Study10.1038/s41746-018-0024-6 ↗