Enhancing Emotion Recognition through Federated Learning: A Multimodal Approach with Convolutional Neural Networks
Source abstract
Abstract
Human–machine interaction covers a range of applications in which machines should understand humans’ commands and predict their behavior. Humans commonly change their mood over time, which affects the way we interact, particularly by changing speech style and facial expressions. As interaction requires quick decisions, low latency is critical for real-time processing. Edge devices, strategically placed near the data source, minimize processing time, enabling real-time decision-making. Edge computing allows us to process data locally, thus reducing the need to send sensitive information further through the network. Despite the wide adoption of audio-only, video-only, and multimodal emotion recognition systems, there is a research gap in terms of analyzing lightweight models and solving privacy challenges to improve model performance. This motivated us to develop a privacy-preserving, lightweight, CNN-based (CNNs are frequently used for processing audio and video modalities) audiovisual emotion recognition model, deployable on constrained edge devices. The model is further paired with a federated learning protocol to preserve the privacy of local clients on edge devices and improve detection accuracy. The results show that the adoption of federated learning improved classification accuracy by ~2%, as well as that the proposed federated learning-based model provides competitive performance compared to other baseline audiovisual emotion recognition models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Music and Audio Processing · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Technical Sciences, University of Novi Sad, 21000 Novi Sad, Serbia
Faculty of Sciences, University of Novi Sad, 21000 Novi Sad, Serbia
Funding
European Union’s Horizon 2020 research
References (39)
Show source-provided references
1. Automatic speech recognition: A survey10.1007/s11042-020-10073-7 ↗
2. Integrating face and voice in person perception10.1016/j.tics.2007.10.001 ↗
3. Survey on audiovisual emotion recognition: Databases, features, and data fusion strategies10.1017/ATSIP.2014.11 ↗
4. Audiovisual emotion recognition in wild10.1007/s00138-018-0960-9 ↗
5. Leveraging recent advances in deep learning for audio-visual emotion recognition10.1016/j.patrec.2021.03.007 ↗
6. Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers10.1016/j.specom.2019.12.001 ↗
7. Speech Emotion Recognition Using Deep Learning Techniques: A Review10.1109/ACCESS.2019.2936124 ↗
8. Hybrid LSTM-transformer model for emotion recognition from speech audio files10.1109/ACCESS.2022.3163856 ↗
9. Dominic Enriquez, M., Rudolf Lucas, C., and Aquino, A. (2023, January 13–14). Scalogram vs Spectrogram as Speech Representation Inputs for Speech Emotion Recognition Using CNN. Proceedings of the 34th Irish Signals and Systems Conference (ISSC), Dublin, Ireland.10.1109/ISSC59246.2023.10162085 ↗
10. A survey on facial emotion recognition techniques: A state-of-the-art literature review10.1016/j.ins.2021.10.005 ↗
11. Gauging human visual interest using multiscale entropy analysis of EEG signals10.1007/s12652-020-02381-5 ↗
12. A practical survey on faster and lighter transformers10.1145/3586074 ↗
13. Jelčicová, Z., and Verhelst, M. (2022). Delta keyword transformer: Bringing transformers to the edge through dynamically pruned multi-head self-attention. arXiv.
14. Bajovic, D., Bakhtiarnia, A., Bravos, G., Brutti, A., Burkhardt, F., Cauchi, D., Chazapis, A., Cianco, C., Dall’Asen, N., and Delic, V. (2021, January 20–22). MARVEL: Multimodal Extreme Scale Data Analytics for Smart Cities Environments. Proceedings of the International Balkan Conference on Communications and Networking (BalkanCom), Novi Sad, Serbia.10.1109/BalkanCom53780.2021.9593258 ↗
15. Ma, F., Zhang, W., Li, Y., Huang, S.-L., and Zhang, L. (2019, January 8–12). An End-to-End Learning Approach for Multimodal Emotion Recognition: Extracting Common and Private Information. Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China.10.1109/ICME.2019.00200 ↗
16. Multimodal emotion recognition from facial expression and speech based on feature fusion10.1007/s11042-022-14185-0 ↗
17. Metric Learning-Based Multimodal Audio-Visual Emotion Recognition
18. C-GCN: Correlation Based Graph Convolutional Network for Audio-Video Emotion Recognition10.1109/TMM.2020.3032037 ↗
19. Fusion of deep learning features with mixture of brain emotional learning for audio-visual emotion recognition10.1016/j.specom.2020.12.001 ↗
20. Federated Learning Meets Human Emotions: A Decentralized Framework for Human–Computer Interaction for IoT Applications10.1109/JIOT.2020.3037207 ↗
21. A federated learning method for real-time emotion state classification from multi-modal streaming10.1016/j.ymeth.2022.03.005 ↗
22. Salman, A., and Busso, C. (2022, January 7–11). Privacy Preserving Personalization for Video Facial Expression Recognition Using Federated Learning. Proceedings of the ICMI ‘22: 2022 International Conference on Multimodal Interaction, Bangalor, India.10.1145/3536221.3556614 ↗
23. Chang, Y., Laridi, S., Ren, Z., Palmer, G., Schuller, B.W., and Fisichella, M. (2022). Robust Federated Learning Against Adversarial Attacks for Speech Emotion Recognition. arXiv.
24. Zhang, T., Feng, T., Alam, S., Lee, S., Zhang, M., Narayanan, S.S., and Avestimehr, S. (2023, January 4–10). FedAudio: A Federated Learning Benchmark for Audio Tasks. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096500 ↗
25. Comparison of speaker dependent and speaker independent emotion recognition10.2478/amcs-2013-0060 ↗
26. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–23). MobileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
27. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
28. User-generated video emotion recognition based on key frames10.1007/s11042-020-10203-1 ↗
29. Emotion recognition using deep learning approach from audio–visual emotional big data10.1016/j.inffus.2018.09.008 ↗
30. Simic, N., Suzic, S., Nosek, T., Vujovic, M., Peric, Z., Savic, M., and Delic, V. (2022). Speaker Recognition Using Constrained Convolutional Neural Networks in Emotional Speech. Entropy, 24.10.3390/e24030414 ↗
31. Speech emotion recognition using machine learning—A systematic review
32. Voxceleb: Large-scale speaker verification in the wild10.1016/j.csl.2019.101027 ↗
33. Simic, N., Suzic, S., Nosek, T., Vujovic, M., and Secujski, M. (September, January 29). Impact of different voting strategies in CNN based speech emotion recognition. Proceedings of the 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia.
34. Hershey, S., Chaudhur, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., and Seybold, B. (2017, January 5–9). CNN architectures for large-scale audio classification. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952132 ↗
35. Bebis, G., Yin, Z., Kim, E., Bender, J., Subr, K., Kwon, B.C., Zhao, J., Kalkofen, D., and Baciu, G. (2020). Advances in Visual Computing. ISVC 2020. Lecture Notes in Computer Science, Springer.10.1007/978-3-030-64559-5 ↗
36. Brecko, A., Kajati, E., Koziorek, J., and Zolotova, I. (2022). Federated Learning for Edge Computing: A Survey. Appl. Sci., 12.10.3390/app12189124 ↗
37. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 20–22). Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54, Ft. Lauderdale, FL, USA.
38. Martin, O., Kotsia, I., Macq, B., and Pitas, I. (2006, January 3–7). The eNTERFACE’ 05 Audio-Visual Emotion Database. Proceedings of the 22nd International Conference on Data Engineering Workshops (ICDEW’06), Atlanta, GA, USA.10.1109/ICDEW.2006.145 ↗
39. Emotion semantics show both cultural variation and universal structure10.1126/science.aaw8160 ↗