Multimodal Fusion with Dual-Attention Based on Textual Double-Embedding Networks for Rumor Detection
Source abstract
Abstract
Rumors may bring a negative impact on social life, and compared with pure textual rumors, online rumors with multiple modalities at the same time are more likely to mislead users and spread, so multimodal rumor detection cannot be ignored. Current detection methods for multimodal rumors do not focus on the fusion of text and picture-region object features, so we propose a multimodal fusion neural network TDEDA (dual-attention based on textual double embedding) applied to rumor detection, which performs a high-level information interaction at the text–image object level and captures visual features associated with keywords using an attention mechanism. In this way, we explored the ability to enhance feature representation with assistance from different modalities in rumor detection, as well as to capture the correlations of the dense interaction between images and text. We conducted comparative experiments on two multimodal rumor detection datasets. The experimental results showed that TDEDA could reasonably handle multimodal information and thus improve the accuracy of rumor detection compared with currently relevant multimodal rumor detection methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Misinformation and Its Impacts · Topic Modeling · Data-Driven Disease Surveillance
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Xinjiang Multilingual Information Technology Laboratory, Xinjiang Multilingual Information Technology Research Center, College of Information Science and Engineering, Xinjiang University, Urumqi 830017, China
School of Software, Xinjiang University, Urumqi 830017, China
Funding
National Natural Science Foundation of China
References (36)
Show source-provided references
1. Wu, K., Yang, S., and Zhu, K.Q. (2015, January 13–17). False rumors detection on sina weibo by propagation structures. Proceedings of the 2015 IEEE 31st International Conference on Data Engineering, Seoul, Republic of Korea.10.1109/ICDE.2015.7113322 ↗
2. Jin, Z., Cao, J., Guo, H., Zhang, Y., and Luo, J. (2017, January 23–27). Multimodal fusion with recur rent neural networks for rumor detection on microblogs. Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, CA, USA.10.1145/3123266.3123454 ↗
3. Zaremba, W., Sutskever, I., and Vinvals, O. (2014). Recurrent neural network regularization. arXiv.
4. Wang, Y., Ma, F., Jin, Z., Yuan, Y., Xun, G., Jha, K., Su, L., and Gao, J. (2018, January 19–23). EANN: Event adversarial neural networks for multi-modal fake news detection. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219903 ↗
5. Khattar, D., Goud, J.S., Gupta, M., and Varma, V. (2019, January 13–17). MVAE: Multimodal variational autoencoder for fake news detection. Proceedings of the 10 Computer Engineering and Applications World Wide Web Conference, San Francisco, CA, USA.10.1145/3308558.3313552 ↗
6. Zhou, X., Wu, J., and Zafarani, R. (2020, January 11–14). Similarity-Aware Multi-modal Fake News Detection. Proceedings of the Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, PAKDD 2020, Singapore. Proceedings, Part II.10.1007/978-3-030-47436-2_27 ↗
7. A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks10.1016/j.ipm.2020.102437 ↗
8. Chen, Y., Li, D., Zhang, P., Sui, J., Lv, Q., Tun, L., and Shang, L. (2022, January 25–29). Cross-modal ambiguity learning for multimodal fake news detection. Proceedings of the ACM Web Conference 2022, Lyon, France.10.1145/3485447.3511968 ↗
9. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep dual transformers for language understanding. arXiv.
10. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
11. Zhu, Y., Wang, X., Zhong, E., Liu, N., Li, H., and Yang, Q. (2012, January 22–26). Discovering spammers in social networks. Proceedings of the AAAI Conference on Artificial Intelligence, Toronto, ON, Canada.
12. Rubin, V.L., Conroy, N., Chen, Y., and Cornwell, S. (2016, January 17). Fake news or truth? using satirical cues to detect potentially misleading news. Proceedings of the Second Workshop on Computational Approaches to Deception Detection, San Diego, CA, USA.10.18653/v1/W16-0802 ↗
13. Castillo, C., Mendoza, M., and Poblete, B. (April, January 28). Information credibility on twitter. Proceedings of the 20th International Conference on World Wide Web, Hyderabad, India.
14. Qazvinian, V., Rosengren, E., Radev, D., and Mei, Q. (2011, January 27–31). Rumor has it: Identifying misinformation in microblogs. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Scotland, UK.
15. Kwon, S., Cha, M., Jung, K., Chen, W., and Wang, Y. (2013, January 7–13). Prominent features of rumor propagation in online social media. Proceedings of the 2013 IEEE 13th International Conference on Data Mining, Dallas, TX, USA.10.1109/ICDM.2013.61 ↗
16. Ma, J., Gao, W., Mitra, P., Kwon, S., Jansen, B.J., Wong, K.F., and Cha, M. (2016, January 9–15). Detecting rumors from microblogs with recurrent neural networks. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16), New York, NY, USA.
17. Chen, T., Li, X., Yin, H., and Zhang, J. (2018, January 3). Call attention to rumors: Deep attention based recurrent neural networks for early rumor detection. Proceedings of the Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2018 Workshops, BDASC, BDM, ML4Cyber, PAISI, DaMEMO, Melbourne, Australia. Revised Selected Papers 22.10.1007/978-3-030-04503-6_4 ↗
18. Yu, F., Liu, Q., Wu, S., Wang, L., and Tan, T. (2017, January 19–25). A Convolutional Approach for Misinformation Identification. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne, Australia.10.24963/ijcai.2017/545 ↗
19. Ma, J., Gao, W., and Wong, K.F. (2018, January 15–20). Rumor detection on twitter with tree-structured recursive neural networks. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1184 ↗
20. Zhang, J., Dong, B., and Philip, S.Y. (2019, January 9–12). Deep diffusive neural network based fake news detection from heterogeneous social networks. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.10.1109/BigData47090.2019.9005556 ↗
21. Liu, Y., and Wu, Y.F. (2018, January 2–7). Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11268 ↗
22. Novel visual and statistical image features for microblogs news verification10.1109/TMM.2016.2617078 ↗
23. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.P. (2017). Tensor fusion network for multimodal sentiment analysis. arXiv.10.18653/v1/D17-1115 ↗
24. Deep multimodal multilinear fusion with high-order polynomial pooling
25. Xu, N., Mao, W., and Chen, G. (February, January 27). Multi-interactive memory network for aspect based multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
26. Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S.C., Wang, X., and Li, H. (2019, January 15–20). Dynamic fusion with intra-and inter-modality attention flow for visual question answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00680 ↗
27. Learning deep architectures for AI10.1561/2200000006 ↗
28. Yang, Z., He, X., Gao, J., Deng, L., and Smola, A. (2016, January 27–30). Stacked attention networks for image question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.10 ↗
29. Liu, Y., Zhang, X., Huang, F., and Li, Z. (2018, January 22–26). Adversarial learning of answer-related representation for visual question answering. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, Torino, Italy.10.1145/3269206.3271765 ↗
30. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
31. Ke, G., He, D., and Liu, T.Y. (2020). Rethinking positional encoding in language pre-training. arXiv.
32. Lu, J., Batra, D., Parikh, D., and Lee, S. (2019, January 8–14). Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
33. Verifying multimedia use at mediaeval 2015
34. Ma, J., Gao, W., Wei, Z., Lu, Y., and Wong, K.F. (2015, January 19–23). Detect rumors using time series of social context information on microblogging websites. Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, Melbourne, Australia.10.1145/2806416.2806607 ↗
35. Yao, L., Mao, C., and Luo, Y. (February, January 27). Graph convolutional networks for text classification. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
36. Godbole, S., and Sarawagi, S. (2004, January 26–28). Discriminative methods for multi-labeled classification. Proceedings of the Advances in Knowledge Discovery and Data Mining: 8th Pacific-Asia Conference, PAKDD 2004, Sydney, Australia. Proceedings 8.