Enhanced Emotion Recognition Through Dynamic Restrained Adaptive Loss and Extended Multimodal Bottleneck Transformer
Source abstract
Abstract
Emotion recognition in video aims to estimate human emotions using acoustic, visual, and linguistic information. This problem is considered multimodal and requires learning different modalities, such as visual, verbal, and vocal cues. Although previous studies have focused on developing sophisticated deep learning models, this work proposes a different approach using dynamic restrained adaptive loss inspired by multitask learning to understand multimodal inputs jointly. This training strategy allows predictions from one modality to enhance the accuracy of predictions from other modalities, mirroring the concept of multitask learning, where the results of one task can improve the performance of related tasks. Furthermore, this work introduces the extended multimodal bottleneck transformer, an efficient and effective mid-fusion method designed for problems involving more than two modalities to enhance the performance of emotion recognition systems. The proposed method significantly improves results compared to other end-to-end multimodal fusion techniques on three multimodal benchmarks—Interactive Emotional Dyadic Motion Capture (IEMOCAP), Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI), and the Chinese Multimodal Sentiment Analysis dataset with independent unimodal annotations (CH-SIMS).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Music and Audio Processing · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence Convergence, Chonnam National University, Gwangju 61186, Republic of Korea
Funding
Institute of Information & Communications Technology Planning & Evaluation(IITP)-Innovative Human Resource Development for Local Intellectualization · National Research Foundation of Korea (NRF) · Institute of Information & communications Technology Planning & Evaluation (IITP)
References (37)
Show source-provided references
1. Dai, W., Cahyawijaya, S., Liu, Z., and Fung, P. (2021, January 6–11). Multimodal End-to-End Sparse Model for Emotion Recognition. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.417 ↗
2. Xia, X., Zhao, Y., and Jiang, D. (2022). Multimodal Interaction Enhanced Representation Learning for Video Emotion Recognition. Front. Neurosci., 16.10.3389/fnins.2022.1086380 ↗
3. Multimodal emotion recognition with deep learning: Advancements, challenges, and future directions10.1016/j.inffus.2023.102218 ↗
4. Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. (November, January 27). CcNet: Criss-Cross Attention for Semantic Segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
5. Hazarika, D., Zimmermann, R., and Poria, S. (2020, January 12–16). MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.
6. Lim, E., Yang, H.J., Yang, H.J., Kim, S.H., Kim, S., Shin, J.E., and Kim, A. (2024, January 18). Modality Weights Based Fusion Model for Social Perception Prediction in Video, Audio, and Text. Proceedings of the 5th on Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Melbourne, VIC, Australia.10.1145/3689062.3689085 ↗
7. Dong, Z., Hu, C., Zhu, L., Ji, X., and Lai, C.S. (2025). A Dual-Pathway Driver Emotion Classification Network Using Multi-Task Learning Strategy: A Joint Verification. IEEE Internet Things J., 10833842.10.1109/JIOT.2025.3527206 ↗
8. Attention Bottlenecks for Multimodal Fusion
9. Hershey, S., Chaudhuri, S., Ellis, D.P.W., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., and Seybold, B. (2017, January 5–9). CNN Architectures for Large-Scale Audio Classification. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952132 ↗
10. Dai, W., Liu, Z., Yu, T., and Fung, P. (2020). Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition. arXiv.10.18653/v1/2020.aacl-main.30 ↗
11. Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., and Yang, K. (2020, January 5–10). CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-Grained Annotation of Modality. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual Event.10.18653/v1/2020.acl-main.343 ↗
12. Multi-Label Multimodal Emotion Recognition with Transformer-Based Fusion and Emotion-Level Representation Learning10.1109/ACCESS.2023.3244390 ↗
13. Zhang, Y., Chen, M., Shen, J., and Wang, C. (March, January 22). Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
14. Tsai, Y.-H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.-P., and Salakhutdinov, R. (August, January 28). Multimodal Transformer for Unaligned Multimodal Language Sequences. Proceedings of the Conference of the Association for Computational Linguistics, Florence, Italy.
15. Cross-Modal Dynamic Convolution for Multi-Modal Emotion Recognition10.1016/j.jvcir.2021.103178 ↗
16. Multi-Channel Weight-Sharing Autoencoder Based on Cascade Multi-Head Attention for Multimodal Emotion Recognition10.1109/TMM.2022.3144885 ↗
17. Liu, Y., Li, S., Wu, Y., Chen, C.-W., Shan, Y., and Qie, X. (2022, January 19–24). UMT: Unified Multi-Modal Transformers for Joint Video Moment Retrieval and Highlight Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00305 ↗
18. Fu, Z., Liu, F., Xu, Q., Qi, J., Fu, X., Zhou, A., and Li, Z. (2022, January 18–22). NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan.10.1109/ICME52920.2022.9859836 ↗
19. Kendall, A., Gal, Y., and Cipolla, R. (2018, January 18–22). Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.
20. End-to-End Learning for Multimodal Emotion Recognition in Video with Adaptive Loss10.1109/MMUL.2021.3080305 ↗
21. Yu, W., Xu, H., Yuan, Z., and Wu, J. (2021, January 2–9). Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event.10.1609/aaai.v35i12.17289 ↗
22. Multimodal Machine Learning: A Survey and Taxonomy
23. Meshgi, K., Mirzaei, M.S., and Sekine, S. (2022, January 10–14). Uncertainty Regularized Multi-Task Learning. Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, Virtual Event.10.18653/v1/2022.wassa-1.8 ↗
24. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. Adv. Neural Inf. Process. Syst., 30.
25. Song, M., Yang, Z., Triantafyllopoulos, A., Jing, X., Karas, V., Jiangjian, X., Zhang, Z., Yoshiharu, Y., and Schuller, B.W. (2022). Dynamic Restrained Uncertainty Weighting Loss for Multitask Learning of Vocal Expression. arXiv.
26. Liu, L., Li, Y., Kuang, Z., Xue, J., Chen, Y., Yang, W., Liao, Q., and Zhang, W. (2021, January 3–7). Towards Impartial Multi-Task Learning. Proceedings of the International Conference on Learning Representations, Virtual Event.
27. IEMOCAP: Interactive Emotional Dyadic Motion Capture Database10.1007/s10579-008-9076-6 ↗
28. Zadeh, A., Bagher, A., Liang, P.P., Poria, S., Cambria, E., and Morency, L.-P. (2018, January 15–20). Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, VIC, Australia. Long Papers.
29. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., and Morency, L.-P. (2018, January 15–20). Efficient Low-Rank Multimodal Fusion with Modality-Specific Factors. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, VIC, Australia. Long Papers.10.18653/v1/P18-1209 ↗
30. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks10.1109/LSP.2016.2603342 ↗
31. Savchenko, A.V. (2021, January 16–18). Facial Expression and Attributes Recognition Based on Multi-Task Learning of Lightweight Neural Networks. Proceedings of the 2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY), Subotica, Serbia.10.1109/SISY52375.2021.9582508 ↗
32. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020, January 26–30). ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
33. Video Sentiment Analysis with Bimodal Information-Augmented Multi-Head Attention10.1016/j.knosys.2021.107676 ↗
34. Sun, Z., Sarma, P., Sethares, W., and Liang, Y. (2020, January 7–12). Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6431 ↗
35. Williams, J., Kleinegesse, S., Comanescu, R., and Radu, O. (2018, January 20–26). Recognizing Emotions in Video Using Multimodal DNN Feature Fusion. Proceedings of the Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), Melbourne, VIC, Australia.10.18653/v1/W18-3302 ↗
36. Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.-P. (2018, January 2–7). Memory Fusion Network for Multi-View Sequential Learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12021 ↗
37. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. (2017, January 7–11). Tensor Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1115 ↗