LGCDF: Label-Guided Contrastive Disentanglement Fusion of Sensitive Attribute-Free Representations for Fair Multimodal Sentiment Analysis
Source abstract
Abstract
Multimodal sentiment analysis (MSA) has emerged as a prominent research frontier, enabling a comprehensive understanding of complex human emotions through the synergistic integration of heterogeneous multimodal signals. However, most existing approaches rely on idealized signal distribution assumptions, overlooking the detrimental impact of demographic bias on representation fairness and fusion robustness. This paper proposes a Label-Guided Contrastive Decoupling Fusion (LGCDF) framework that enhances model robustness to demographic bias by learning and fusing multimodal representations invariant to Sensitive Attributes (SAs). Specifically, the proposed LGCDF framework employs gender-sensitive attribute information as modality-level constraints to achieve language-centric cross-modal sentiment alignment, which is accomplished by computing contrastive losses between text–audio and text–visual feature pairs. Moreover, it introduces a SA-guided contrastive decoupling mechanism that decomposes multimodal representations into SA-related and -independent components. The SA-independent components are subsequently fused through a cross-modal attention fusion strategy, thereby facilitating fair sentiment representation and enabling efficient and robust multimodal information fusion. Extensive experimental results demonstrate that the proposed LGCDF framework achieves superior performance in fair representation learning and cross-modal information fusion while maintaining strong robustness under varying gender distribution biases.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Sentiment Analysis and Opinion Mining · Generative Adversarial Networks and Image Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Eastern Institute of Technology, Ningbo 315200, China
School of Computer Science and Technology, University of Science and Technology of China, Hefei 230052, China
School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, China
Funding
2035 Key Research and Development Program of Ningbo City
References (66)
Show source-provided references
1. A survey of cutting-edge multimodal sentiment analysis
2. M2lens: Visualizing and explaining multimodal models for sentiment analysis10.1109/TVCG.2021.3114794 ↗
3. MuSe-CarASTE: A comprehensive dataset for aspect sentiment triplet extraction in automotive review videos10.1016/j.eswa.2024.125695 ↗
4. MEmoR: A multimodal emotion recognition using affective biomarkers for smart prediction of emotional health for people analytics in smart industries10.1016/j.imavis.2022.104483 ↗
5. Brody, L.R. (2013). On understanding gender differences in the expression of emotion: Gender roles, socialization, and language. Human Feelings, Routledge.
6. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., and Morency, L.P. (2018). Efficient low-rank multimodal fusion with modality-specific factors. arXiv.10.18653/v1/P18-1209 ↗
7. Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.P. (2018, January 2–7). Memory fusion network for multi-view sequential learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12021 ↗
8. A survey on bias and fairness in machine learning10.1145/3457607 ↗
9. Adewumi, T., Alkhaled, L., Gurung, N., van Boven, G., and Pagliai, I. (2024). Fairness and bias in multimodal ai: A survey. arXiv.
10. Satar, B., Zhu, H., Zhang, H., and Lim, J.H. (2023). Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention. arXiv.
11. Sharma, S., Zhang, Y., Ríos Aliaga, J.M., Bouneffouf, D., Muthusamy, V., and Varshney, K.R. (2020, January 7–8). Data augmentation for discrimination prevention and bias disambiguation. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA.10.1145/3375627.3375865 ↗
12. Wang, P., Zhou, Q., Wu, Y., Chen, T., and Hu, J. (March, January 25). DLF: Disentangled-language-focused multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
13. Lim, J., Kim, Y., Kim, B., Ahn, C., Shin, J., Yang, E., and Han, S. (2023, January 18–22). Biasadv: Bias-adversarial augmentation for model debiasing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00373 ↗
14. The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective
15. Kazhdan, D., Dimanov, B., Terre, H.A., Jamnik, M., Liò, P., and Weller, A. (2021). Is disentanglement all you need? comparing concept-based & disentanglement approaches. arXiv.
16. Accuracy and fairness trade-offs in machine learning: A stochastic multi-objective approach10.1007/s10287-022-00425-z ↗
17. Wang, Y., Wang, X., Beutel, A., Prost, F., Chen, J., and Chi, E.H. (2021, January 14–18). Understanding and improving fairness-accuracy trade-offs in multi-task learning. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Virtual.10.1145/3447548.3467326 ↗
18. Schmitz, M., Ahmed, R., and Cao, J. (2022). Bias and fairness on multimodal emotion detection algorithms. arXiv.
19. Yang, D., Li, M., Xiao, D., Liu, Y., Yang, K., Chen, Z., Wang, Y., Zhai, P., Li, K., and Zhang, L. (2025). Towards multimodal sentiment analysis debiasing via bias purification. Proceedings of the European Conference on Computer Vision, Milan, Italy, 29 September–4 October 2024, Springer.10.1007/978-3-031-73636-0_27 ↗
20. Diversity and balance: Multimodal sentiment analysis using multimodal-prefixed and cross-modal attention10.1109/TAFFC.2024.3430045 ↗
21. Metrics for dataset demographic bias: A case study on facial expression recognition10.1109/TPAMI.2024.3361979 ↗
22. Fairbalance: How to achieve equalized odds with data pre-processing10.1109/TSE.2024.3431445 ↗
23. A review on fairness in machine learning10.1145/3494672 ↗
24. Optimizing fairness and accuracy: A Pareto optimal approach for decision-making10.1007/s43681-024-00508-4 ↗
25. Birhane, A., Prabhu, V.U., and Kahembwe, E. (2021). Multimodal datasets: Misogyny, pornography, and malignant stereotypes. arXiv.
26. Towards measuring fairness in ai: The casual conversations dataset10.1109/TBIOM.2021.3132237 ↗
27. Mitigating demographic bias in facial datasets with style-based multi-attribute transfer10.1007/s11263-021-01448-w ↗
28. Toward robust incomplete multimodal sentiment analysis via hierarchical representation learning
29. Trusted Multimodal Socio-Cyber Sentiment Analysis Based on Disentangled Hierarchical Representation Learning10.1109/TCSS.2023.3324462 ↗
30. Sun, T., Ni, J., Wang, W., Jing, L., Wei, Y., and Nie, L. (November, January 29). General debiasing for multimodal sentiment analysis. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
31. MulDeF: A Model-Agnostic Debiasing Framework for Robust Multimodal Sentiment Analysis10.1109/TMM.2024.3521836 ↗
32. Domain adaptation for bias mitigation in affective computing: Use cases for facial emotion recognition and sentiment analysis systems10.1007/s42452-025-06659-1 ↗
33. A multimodal approach to cross-lingual sentiment analysis with ensemble of transformer and LLM10.1038/s41598-024-60210-7 ↗
34. Mitigate gender bias using negative multi-task learning10.1007/s11063-023-11368-0 ↗
35. Lin, J., Li, Q., Xie, G., Guan, Z., Jiang, Y., Xu, T., Zhang, Z., and Zhao, P. (November, January 28). Mitigating Sample Selection Bias with Robust Domain Adaption in Multimedia Recommendation. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, VIC, Australia.
36. A Multimodal Semantic Fusion Network with Cross-Modal Alignment for Multimodal Sentiment Analysis10.1145/3744648 ↗
37. Yang, J., Yu, Y., Niu, D., Guo, W., and Xu, Y. (2023). Confede: Contrastive feature decomposition for multimodal sentiment analysis. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics.10.18653/v1/2023.acl-long.421 ↗
38. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis10.1109/TAFFC.2022.3172360 ↗
39. Modeling intra-and inter-modal relations: Hierarchical graph contrastive learning for multimodal sentiment analysis
40. Shou, Y., Meng, T., Ai, W., Yin, N., and Li, K. (2024). Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition. arXiv.
41. Towards robust sentiment analysis with multimodal interaction graph and hybrid contrastive learning10.1016/j.patcog.2025.111870 ↗
42. Gao, Y., Zhang, S., Sun, J., Yu, S.S., and Yoshii, A. (2024, January 15–17). Latent bias mitigation via contrastive learning. Proceedings of the 2024 5th International Conference on Artificial Intelligence in Electronics Engineering, Phuket, Thailand.10.1145/3658835.3658843 ↗
43. Yang, Y., Nushi, B., Palangi, H., and Mirzasoleiman, B. (2023, January 23–29). Mitigating spurious correlations in multi-modal models during fine-tuning. Proceedings of the International Conference on Machine Learning. PMLR, Honolulu, HI, USA.
44. Disentanglement translation network for multimodal sentiment analysis10.1016/j.inffus.2023.102031 ↗
45. Toward Robust Multimodal Sentiment Analysis using multimodal foundational models10.1016/j.eswa.2025.126974 ↗
46. SDRS: Sentiment-aware disentangled representation shifting for multimodal sentiment analysis10.1109/TAFFC.2025.3539225 ↗
47. A multi-scale representation and multi-level decision learning network for multimodal sentiment analysis10.1016/j.eswa.2025.129341 ↗
48. Exploring explicitly disentangled features for domain generalization10.1109/TCSVT.2023.3269534 ↗
49. Learning debiased and disentangled representations for semantic segmentation
50. Fine-grained Semantic Disentanglement Network for Multimodal Sarcasm Analysis10.1145/3722558 ↗
51. Zhu, Y., Li, J., Zheng, Z., and Chen, L. (2024, January 13–17). Fair graph representation learning via sensitive attribute disentanglement. Proceedings of the ACM Web Conference 2024, Singapore.10.1145/3589334.3645532 ↗
52. Disentangled variational auto-encoder for multimodal fusion performance analysis in multimodal sentiment analysis10.1016/j.knosys.2024.112372 ↗
53. Disentangled variational auto-encoder enhanced by counterfactual data for debiasing recommendation10.1007/s40747-023-01314-x ↗
54. Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages10.1109/MIS.2016.94 ↗
55. Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., and Morency, L.P. (2018). Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics.
56. Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., and Yang, K. (2020, January 5–10). CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.343 ↗
57. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.P. (2017). Tensor fusion network for multimodal sentiment analysis. arXiv.10.18653/v1/D17-1115 ↗
58. Cambria, E., Hazarika, D., Poria, S., Hussain, A., and Subramanyam, R. (2017). Benchmarking multimodal sentiment analysis. Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing, Springer.10.1007/978-3-319-77116-8_13 ↗
59. Tsai, Y.H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.P., and Salakhutdinov, R. (2019). Multimodal Transformer for Unaligned Multimodal Language Sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence, Italy, Association for Computational Linguistics.10.18653/v1/P19-1656 ↗
60. Hazarika, D., Zimmermann, R., and Poria, S. (2020). MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis. arXiv.10.1145/3394171.3413678 ↗
61. Tsai, Y.H., Liang, P.P., Zadeh, A., Morency, L., and Salakhutdinov, R. (2019, January 6–9). Learning Factorized Multimodal Representations. Proceedings of the ICLR, New Orleans, LA, USA.
62. Han, W., Chen, H., and Poria, S. (2021, January 7–11). Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.723 ↗
63. Zhu, A., Hu, M., Wang, X., Yang, J., Tang, Y., and Ren, F. (November, January 28). KEBR: Knowledge Enhanced Self-Supervised Balanced Representation for Multimodal Sentiment Analysis. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, VIC, Australia.
64. Yu, W., Xu, H., Yuan, Z., and Wu, J. (2021, January 2–9). Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i12.17289 ↗
65. Mao, H., Yuan, Z., Xu, H., Yu, W., Liu, Y., and Gao, K. (2022). M-SENA: An Integrated Platform for Multimodal Sentiment Analysis. arXiv.10.18653/v1/2022.acl-demo.20 ↗
66. Investigating fairness in machine learning-based audio sentiment analysis10.1007/s43681-024-00453-2 ↗