M2ER: Multimodal Emotion Recognition Based on Multi-Party Dialogue Scenarios
Source abstract
Abstract
Researchers have recently focused on multimodal emotion recognition, but issues persist in recognizing emotions in multi-party dialogue scenarios. Most studies have only used text and audio modality, ignoring the video modality. To address this, we propose M2ER, a multimodal emotion recognition scheme based on multi-party dialogue scenarios. Addressing the issue of multiple faces appearing in the same frame of the video modality, M2ER introduces a method using multi-face localization for speaker recognition to eliminate the interference of non-speakers. The attention mechanism is used to fuse and classify different modalities. We conducted extensive experiments in unimodal and multimodal fusion using the multi-party dialogue dataset MELD. The results show that M2ER achieves superior emotion recognition in both text and audio modalities compared to the baseline model. The proposed method using speaker recognition in the video modality improves emotion recognition performance by 6.58% compared to the method without speaker recognition. In addition, the multimodal fusion based on the attention mechanism also outperforms the baseline fusion model.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Sentiment Analysis and Opinion Mining · Speech and dialogue systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Communication Engineering, Communication University of China, Dingfuzhuang, Chaoyang District, Beijing 10024, China
School of Computing, Newcastle University, Newcastle upon Tyne NE1 7RU, UK
References (50)
Show source-provided references
1. An argument for basic emotions10.1080/02699939208411068 ↗
2. Constants across cultures in the face and emotion10.1037/h0030377 ↗
3. Picard, R.W. (2000). Affective Computing, MIT Press.10.7551/mitpress/1140.001.0001 ↗
4. Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M.A., Schuller, B., and Zafeiriou, S. (2016, January 20–25). Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7472669 ↗
5. Learning affective features with a hybrid deep model for audio-visual emotion recognition10.1109/TCSVT.2017.2719043 ↗
6. Facial expression recognition in videos using dynamic kernels10.1109/TIP.2020.3011846 ↗
7. Affectnet: A database for facial expression, valence, and arousal computing in the wild10.1109/TAFFC.2017.2740923 ↗
8. Mao, J., Xu, R., Yin, X., Chang, Y., Nie, B., and Huang, A. (2023). Poster v2: A simpler and stronger facial expression recognition network. arXiv.
9. Panagiotis, A., Filntisis, P.P., and Maragos, P. (2021). Exploiting emotional dependencies with graph convolutional networks for facial expression recognition. arXiv.
10. In search of a robust facial expressions recognition model: A large-scale visual cross-corpus study10.1016/j.neucom.2022.10.013 ↗
11. Bakariya, B., Singh, A., Singh, H., Raju, P., Rajpoot, R., and Mohbey, K.K. (2023). Evolving Systems, Springer.
12. Meena, G., Mohbey, K.K., Indian, A., Khan, M.Z., and Kumar, S. (2023). Multimedia Tools and Applications, Springer.
13. Savchenko, A.V. (2021, January 16–18). Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. Proceedings of the 2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY), Subotica, Serbia.10.1109/SISY52375.2021.9582508 ↗
14. Sentiment analysis on images using convolutional neural networks based inception-v3 transfer learning approach
15. Facial emotion recognition using convolutional neural networks (ferc)10.1007/s42452-020-2234-1 ↗
16. Cnn and lstm based facial expression analysis model for a humanoid robot10.1109/ACCESS.2019.2928364 ↗
17. Cnn-lstm facial expression recognition method fused with two-layer attention mechanism10.1155/2022/7450637 ↗
18. Sang, D.V., and Ha, P.T. (2018, January 5–6). Discriminative deep feature learning for facial emotion recognition. Proceedings of the 2018 1st International Conference on Multimedia Analysis and Pattern Recognition (MAPR), Ho Chi Minh City, Vietnam.10.1109/MAPR.2018.8337514 ↗
19. Huang, G., Liu, Z., Maaten, L.V.D., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
20. Xue, F., Wang, Q., and Guo, G. (2021, January 11–17). Transfer: Learning relation-aware facial expression representations with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00358 ↗
21. Facial expression recognition with visual transformers and attentional selective fusion10.1109/TAFFC.2021.3122146 ↗
22. Chen, C.-F.R., Fan, Q., and Panda, R. (2021, January 11–17). Crossvit: Cross-attention multi-scale vision transformer for image classification. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00041 ↗
23. Heo, B., Yun, S., Han, D., Chun, S., Choe, J., and Oh, S.J. (2021, January 11–17). Rethinking spatial dimensions of vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01172 ↗
24. Nguyen, D., Nguyen, K., Sridharan, S., Ghasemi, A., Dean, D., and Fookes, C. (2017, January 24–31). Deep spatio-temporal features for multimodal emotion recognition. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.140 ↗
25. Multi-modal emotion recognition by fusing correlation features of speech-visual10.1109/LSP.2021.3055755 ↗
26. Zadeh, A., Zellers, R., Pincus, E., and Morency, L. (2016). MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv.
27. Zadeh, A., Liang, P.P., Poria, S., Cambria, E., and Morency, L.-P. (2018, January 15–20). Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia. Available online: https://api.semanticscholar.org/CorpusID:51868869.
28. Dai, W., Cahyawijaya, S., Liu, Z., and Fung, P. (2021). Multimodal end-to-end sparse model for emotion recognition. arXiv.10.18653/v1/2021.naacl-main.417 ↗
29. Interactive multimodal attention network for emotion recognition in conversation10.1109/LSP.2021.3078698 ↗
30. Khare, A., Parthasarathy, S., and Sundaram, S. (2021, January 19–22). Self-supervised learning with cross-modal transformers for emotion recognition. Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383618 ↗
31. Lv, F., Chen, X., Huang, Y., Duan, L., and Lin, G. (2021, January 20–25). Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00258 ↗
32. Xie, B., Sidulova, M., and Park, C.H. (2021). Robust multimodal emotion recognition from conversation with transformer-based crossmodality fusion. Sensors, 21.10.3390/s21144913 ↗
33. Zhang, L., Liu, C., and Jia, N. (2023). Uni2mul: A conformer-based multimodal emotion classification model by considering unimodal expression differences with multi-task learning. Appl. Sci., 13.10.3390/app13179910 ↗
34. Scanet: Improving multimodal representation and fusion with sparse-and cross-attention for multimodal sentiment analysis10.1002/cav.2090 ↗
35. Han-regru: Hierarchical attention network with residual gated recurrent unit for emotion recognition in conversation10.1007/s00521-020-05063-7 ↗
36. Jiao, W., Lyu, M.R., and King, I. (2019). Real-time emotion recognition via attention gated hierarchical memory network. arXiv.10.1609/aaai.v34i05.6309 ↗
37. Adapted dynamic memory network for emotion recognition in conversation10.1109/TAFFC.2020.3005660 ↗
38. Hu, D., Hou, X., Wei, L., Jiang, L., and Mo, Y. (2022, January 22–27). Mm-dfn: Multimodal dynamic fusion network for emotion recognition in conversations. Proceedings of the ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747397 ↗
39. Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (2018). MELD: A multimodal multi-party dataset for emotion recognition in conversations. arXiv.10.18653/v1/P19-1050 ↗
40. Firdaus, M., Chauhan, H., Ekbal, A., and Bhattacharyya, P. (2020, January 8–13). MEISD: A multimodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain. Available online: https://aclanthology.org/2020.coling-main.393.10.18653/v1/2020.coling-main.393 ↗
41. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
42. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
43. Munikar, M., Shakya, S., and Shrestha, A. (2019, January 5). Fine-grained sentiment classification using bert. Proceedings of the 2019 Artificial Intelligence for Transforming Business and Society (AITB), Kathmandu, Nepal.10.1109/AITB48515.2019.8947435 ↗
44. Tree framework with bert word embedding for the recognition of Chinese implicit discourse relations10.1109/ACCESS.2020.3019500 ↗
45. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
46. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
47. Deep learning based semantic personalized recommendation system
48. Kim, Y. (2014). Convolutional neural networks for sentence classification. arXiv.10.3115/v1/D14-1181 ↗
49. Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., and Morency, L.-P. (August, January 30). Context-dependent sentiment analysis in user-generated videos. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada. Available online: https://aclanthology.org/P17-1081.
50. Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., and Cambria, E. (2018). DialogueRNN: An Attentive RNN for Emotion Detection in Conversations. arXiv.10.1609/aaai.v33i01.33016818 ↗