Face and Voice Recognition-Based Emotion Analysis System (EAS) to Minimize Heterogeneity in the Metaverse
Source abstract
Abstract
The metaverse, where users interact through avatars, is evolving to closely mirror the real world, requiring realistic object responses based on users’ emotions. While technologies like eye-tracking and hand-tracking transfer physical movements into virtual spaces, accurate emotion detection remains challenging. This study proposes the “Face and Voice Recognition-based Emotion Analysis System (EAS)” to bridge this gap, assessing emotions through both voice and facial expressions. EAS utilizes a microphone and camera to gauge emotional states, combining these inputs for a comprehensive analysis. It comprises three neural networks: the Facial Emotion Analysis Model (FEAM), which classifies emotions using facial landmarks; the Voice Sentiment Analysis Model (VSAM), which detects vocal emotions even in noisy environments using MCycleGAN; and the Metaverse Emotion Recognition Model (MERM), which integrates FEAM and VSAM outputs to infer overall emotional states. EAS’s three primary modules—Facial Emotion Recognition, Voice Emotion Recognition, and User Emotion Analysis—analyze facial features and vocal tones to detect emotions, providing a holistic emotional assessment for realistic interactions in the metaverse. The system’s performance is validated through dataset testing, and future directions are suggested based on simulation outcomes.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Education and Learning Interventions · Diverse Topics in Contemporary Research · Generative Adversarial Networks and Image Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Software, College of Engineering, Catholic Kwandong University, Gangneung 25601, Republic of Korea
References (24)
Show source-provided references
1. Study for Classification of Facial Expression using Distance Features of Facial Landmarks
2. Classification of Facial Expressions Using Landmark-based Ensemble Network10.9728/dcs.2022.23.1.117 ↗
3. Facial Landmark Detection by Stacked Hourglass Network with Transposed Convolutional Layer
4. Integral Regression Network for Facial Landmark Detection
5. Sohn, M.-K., Lee, S.-H., and Kim, H. (2021, January 16–18). Analysis and implementation of a deep learning system for face and its landmark detection on mobile applications. Proceedings of the Symposium of the Korean Institute of Communications and Information Sciences, Seoul, Republic of Korea.
6. Music player using emotion classification of facial expressions
7. Facial Expression Classification Using Deep Convolutional Neural Network
8. Choi, W., Kim, T., Bae, H., and Kim, J. (2019). Comparison of Emotion Prediction Performance of CNN Architectures, Korean Institute of Information Scientists and Engineers.
9. A Design of Small Scale Deep CNN Model for Facial Expression Recognition using the Low Resolution Image
10. Park, J.H. (2024, January 09). A Noise Filtering Scheme with Machine Learning for Audio Content Recognition. 2019.02. Available online: https://repository.hanyang.ac.kr/handle/20.500.11754/100017.
11. A Study on a Non-Voice Section Detection Model among Speech Signals using CNN Algorithm
12. Lim, K.-H. (2019, January 18). GAN with Dual Discriminator for Non-stationary Noise Cancellation. Proceedings of the Symposium of the Korean Institute of Information Scientists and Engineers, Republic of Korea.
13. Time-domain Speech Enhancement Using Generative Adversarial Networks10.1016/j.specom.2019.09.001 ↗
14. Michelsanti, D., and Tan, Z.H. (2017). Conditional generative adversarial networks for speech enhancement and noise-robust speaker verification. arXiv.10.21437/Interspeech.2017-1620 ↗
15. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
16. Lai, W.H., Wang, S.L., and Xu, Z.Y. (2022). CycleGAN-Based Singing/Humming to Instrument Conversion Technique. Electronics, 11.10.3390/electronics11111724 ↗
17. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., and Paul Smolley, S. (2017, January 22–29). Least squares generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.304 ↗
18. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
19. Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv.
20. Recognition of a baby’s emotional cry towards robotics baby caregiver10.5772/55406 ↗
21. AI-based Infant State Recognition Using Crying Sound
22. (2024, April 02). Composite Images for Korean Emotion Recognition. Available online: https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=82.
23. (2024, April 02). Conversational Speech Dataset for Sentiment Classification. Available online: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&dataSetSn=263.
24. (2024, April 02). ESC-50: Dataset for Environmental Sound Classification, Noisy Sound Data Set. Available online: https://github.com/karolpiczak/ESC-50.