Applying a Convolutional Vision Transformer for Emotion Recognition in Children with Autism: Fusion of Facial Expressions and Speech Features
Source abstract
Abstract
With advances in digital technology, including deep learning and big data analytics, new methods have been developed for autism diagnosis and intervention. Emotion recognition and the detection of autism in children are prominent subjects in autism research. Typically using single-modal data to analyze the emotional states of children with autism, previous research has found that the accuracy of recognition algorithms must be improved. Our study creates datasets on the facial and speech emotions of children with autism in their natural states. A convolutional vision transformer-based emotion recognition model is constructed for the two distinct datasets. The findings indicate that the model achieves accuracies of 79.12% and 83.47% for facial expression recognition and Mel spectrogram recognition, respectively. Consequently, we propose a multimodal data fusion strategy for emotion recognition and construct a feature fusion model based on an attention mechanism, which attains a recognition accuracy of 90.73%. Ultimately, by using gradient-weighted class activation mapping, a prediction heat map is produced to visualize facial expressions and speech features under four emotional states. This study offers a technical direction for the use of intelligent perception technology in the realm of special education and enriches the theory of emotional intelligence perception of children with autism.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Autism Spectrum Disorder Research · Assistive Technology in Communication and Mobility · Social Robot Interaction and HRI
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Education, Zhejiang University of Technology, Hangzhou 310023, China
Mental Health Education Centre, Zhejiang University of Finance and Economics, Hangzhou 310018, China
Funding
National Natural Science Foundation of China · National Social Science Fund of China · Humanities and Social Sciences Research Projects of the Ministry of Education
References (42)
Show source-provided references
1. Using emotion recognition technologies to teach children with autism spectrum disorder how to identify and express emotions10.1007/s10209-021-00818-y ↗
2. A review of the use of EEG connectivity to measure the neurological characteristics of the sensory features in young people with autism10.1515/revneuro-2018-0070 ↗
3. Shi, J., Liu, C., Ishi, C.T., and Ishiguro, H. (2020). Skeleton-based emotion recognition based on two-stream self-attention enhanced spatial-temporal graph convolutional network. Sensors, 21.10.3390/s21010205 ↗
4. Research on Learning Affective Computing in Online Education: From the Perspective of Multi-source Data Fusion
5. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7–15). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.21236/ADA623249 ↗
6. Facial expression recognition based on deep evolutional spatial-temporal networks10.1109/TIP.2017.2689999 ↗
7. Samad, M.D., Bobzien, J.L., Harrington, J.W., and Iftekharuddin, K.M. (2015, January 9–12). Analysis of facial muscle activation in children with autism using 3D imaging. Proceedings of the 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Washington, DC, USA.10.1109/BIBM.2015.7359704 ↗
8. Metallinou, A., Grossman, R.B., and Narayanan, S. (2013, January 15–19). Quantifying atypicality in affective facial expressions of children with autism spectrum disorders. Proceedings of the 2013 IEEE International Conference on Multimedia and Expo (ICME), San Jose, CA, USA.10.1109/ICME.2013.6607640 ↗
9. A computational study of expressive facial dynamics in children with autism10.1109/TAFFC.2016.2578316 ↗
10. American Psychiatric Association (2013). Diagnostic and Statistical Manual of Mental Disorders: DSM-5, American Psychiatric Association.10.1176/appi.books.9780890425596 ↗
11. Positive, negative, neutral—or unknown? The perceived valence of emotions expressed by young autistic children in a novel context suited to autism10.1177/13623613211068221 ↗
12. Bone, D., Black, M.P., Lee, C.C., Williams, M.E., Levitt, P., Lee, S., and Narayanan, S.S. (2012, January 9–13). Spontaneous-Speech Acoustic-Prosodic Features of Children with Autism and the Interacting Psychologist. Proceedings of the InterSpeech, Portland, OR, USA.10.21437/Interspeech.2012-307 ↗
13. Bone, D., Black, M.P., Ramakrishna, A., Grossman, R.B., and Narayanan, S.S. (2015, January 6–10). Acoustic-prosodic correlates of’awkward’prosody in story retellings from adolescents with autism. Proceedings of the Interspeech, Dresden, Germany.10.21437/Interspeech.2015-374 ↗
14. Acoustic differences in the imitation of prosodic patterns in children with autism spectrum disorders10.1016/j.rasd.2011.03.012 ↗
15. Pre-and paralinguistic vocal production in ASD: Birth through school age10.1007/s11920-019-1113-1 ↗
16. Winczura, Dziecko z autyzmem: Terapia deficytów poznawczych a teoria umysłu
17. Jaklewicz, H. (1993). Autyzm wczesnodzieciÄ™ cy: Diagnoza, przebieg, leczenie, Gdańskie Wydawnictwo Psychologiczne.
18. Mehrabian, A. (2017). Communication without words. Communication Theory, Routledge.10.4324/9781315080918-15 ↗
19. Undifferentiated facial electromyography responses to dynamic, audio-visual emotion displays in individuals with autism spectrum disorders10.1111/desc.12062 ↗
20. A comparative study of Autistic Children Emotion recognition based on Spatio-Temporal and Deep analysis of facial expressions features during a Meltdown Crisis10.1007/s11042-020-09451-y ↗
21. Real-time facial emotion recognition system among children with autism based on deep learning and IoT10.1007/s00521-023-08372-9 ↗
22. Landowska, A., Karpus, A., Zawadzka, T., Robins, B., Erol Barkana, D., Kose, H., Zorcec, T., and Cummins, N. (2022). Automatic emotion recognition in children with autism: A systematic literature review. Sensors, 22.10.3390/s22041649 ↗
23. Assessment on speech emotion recognition for autism spectrum disorder children using support vector machine
24. Towards voice based prediction and analysis of emotions in ASD children10.3233/JIFS-189854 ↗
25. Garg, U., Agarwal, S., Gupta, S., Dutt, R., and Singh, D. (2020, January 25–26). Prediction of emotions from the audio speech signals using MFCC, MEL and Chroma. Proceedings of the 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN), Los Alamitos, CA, USA.10.1109/CICN49253.2020.9242635 ↗
26. Multimodal Emotion Recognition with deep learning: Advancements, challenges, and future directions10.1016/j.inffus.2023.102218 ↗
27. Multimodal multi-channel on-line speaker diarization using sensor fusion through SVM10.1109/TMM.2015.2463722 ↗
28. Deep multimodal fusion for semantic image segmentation: A survey10.1016/j.imavis.2020.104042 ↗
29. Survey of research on multimodal fusion technology for deep learning
30. Dai, Y., Gieseke, F., Oehmcke, S., Wu, Y., and Barnard, K. (2021, January 5–9). Attentional feature fusion. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Virtual.10.1109/WACV48630.2021.00360 ↗
31. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
32. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
33. Oord, A.v.d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv.
34. Li, J., Bhat, A., and Barmaki, R. (2021). A two-stage multi-modal affect analysis framework for children with autism spectrum disorder. arXiv.
35. Reducing facial dynamics’ speed during speech enhances attention to mouth in children with autism spectrum disorder: An eye-tracking study10.1017/S0954579420000292 ↗
36. Wodajo, D., and Atnafu, S. (2021). Deepfake video detection using convolutional vision transformer. arXiv.
37. Speech emotion recognition from 3D log-mel spectrograms with deep learning network10.1109/ACCESS.2019.2938007 ↗
38. Bulatović, N., and Djukanović, S. (2022, January 9–11). Mel-spectrogram features for acoustic vehicle detection and speed estimation. Proceedings of the 2022 26th International Conference on Information Technology (IT), San Carlos, Costa Rica.10.1109/IT54280.2022.9743540 ↗
39. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
40. Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. (2021, January 11–17). Cvt: Introducing convolutions to vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00009 ↗
41. Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jégou, H., and Douze, M. (2021, January 11–17). Levit: A vision transformer in convnet’s clothing for faster inference. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01204 ↗
42. Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., and Shi, H. (2021). Escaping the big data paradigm with compact transformers. arXiv.