FMFN: Fine-Grained Multimodal Fusion Networks for Fake News Detection
Source abstract
Abstract
As one of the most popular social media platforms, microblogs are ideal places for news propagation. In microblogs, tweets with both text and images are more likely to attract attention than text-only tweets. This advantage is exploited by fake news producers to publish fake news, which has a devasting impact on individuals and society. Thus, multimodal fake news detection has attracted the attention of many researchers. For news with text and image, multimodal fake news detection utilizes both text and image information to determine the authenticity of news. Most of the existing methods for multimodal fake news detection obtain a joint representation by simply concatenating a vector representation of the text and a visual representation of the image, which ignores the dependencies between them. Although there are a small number of approaches that use the attention mechanism to fuse them, they are not fine-grained enough in feature fusion. The reason is that, for a given image, there are multiple visual features and certain correlations between these features. They do not use multiple feature vectors representing different visual features to fuse with textual features, and ignore the correlations, resulting in inadequate fusion of textual features and visual features. In this paper, we propose a novel fine-grained multimodal fusion network (FMFN) to fully fuse textual features and visual features for fake news detection. Scaled dot-product attention is utilized to fuse word embeddings of words in the text and multiple feature vectors representing different features of the image, which not only considers the correlations between different visual features but also better captures the dependencies between textual features and visual features. We conduct extensive experiments on a public Weibo dataset. Our approach achieves competitive results compared with other methods for fusing visual representation and text representation, which demonstrates that the joint representation learned by the FMFN (which fuses multiple visual features and multiple textual features) is better than the joint representation obtained by fusing a visual representation and a text representation in determining fake news.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Misinformation and Its Impacts · COVID-19 diagnosis using AI · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Shanghai Key Laboratory of Trustworthy Computing, East China Normal University, Shanghai 200062, China
Funding
National Natural Science Foundation of China · Open Project of International Joint Laboratory of Trustworthy Software, the Key Laboratory of Embedded System and Service Computing Ministry of Education (Tongji University)
References (40)
Show source-provided references
1. News Consuming Habits of Young Social Media Users in the Era of Fake News
2. Helmstetter, S., and Paulheim, H. (2021). Collecting a Large Scale Dataset for Classifying Fake News Tweets Using Weak Supervision. Future Internet, 13.10.3390/fi13050114 ↗
3. Zakharchenko, A., Peráček, T., Fedushko, S., Syerov, Y., and Trach, O. (2021). When Fact-Checking and ‘BBC Standards’ Are Helpless: ‘Fake Newsworthy Event’ Manipulation and the Reaction of the ‘High-Quality Media’ on It. Sustainability, 13.10.3390/su13020573 ↗
4. Jin, Z., Cao, J., Guo, H., Zhang, Y., and Luo, J. (2017, January 23–27). Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs. Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, CA, USA.10.1145/3123266.3123454 ↗
5. Wang, Y., Ma, F., Jin, Z., Yuan, Y., Xun, G., Jha, K., Su, L., and Gao, J. (2018, January 19–23). EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219903 ↗
6. Khattar, D., Goud, J.S., Gupta, M., and Varma, V. (2019, January 13–17). MVAE: Multimodal Variational Autoencoder for Fake News Detection. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.10.1145/3308558.3313552 ↗
7. A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks10.1016/j.ipm.2020.102437 ↗
8. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A Robustly Optimized BERT Pretraining Approach. arXiv.
9. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
10. Islam, N., Shaikh, A., Qaiser, A., Asiri, Y., Almakdi, S., Sulaiman, A., Moazzam, V., and Babar, S.A. (2021). Ternion: An Autonomous Model for Fake News Detection. Appl. Sci., 11.10.3390/app11199292 ↗
11. Alonso, M.A., Vilares, D., Gómez-Rodríguez, C., and Vilares, J. (2021). Sentiment Analysis for Fake News Detection. Electronics, 10.10.3390/electronics10111348 ↗
12. Ma, J., Gao, W., Mitra, P., Kwon, S., Jansen, B.J., Wong, K.-F., and Cha, M. (2016, January 9–15). Detecting rumors from microblogs with recurrent neural networks. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, NY, USA.
13. Yu, F., Liu, Q., Wu, S., Wang, L., and Tan, T. (2017, January 19–25). A convolutional approach for misinformation identification. Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia.10.24963/ijcai.2017/545 ↗
14. Ma, J., Gao, W., and Wong, K.-F. (2019, January 13–17). Detect Rumors on Twitter by Promoting Information Campaigns with Generative Adversarial Learning. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.10.1145/3308558.3313741 ↗
15. Novel Visual and Statistical Image Features for Microblogs News Verification10.1109/TMM.2016.2617078 ↗
16. Fake News Detection on Social Media: A Data Mining Perspective10.1145/3137597.3137600 ↗
17. Qi, P., Cao, J., Yang, T., Guo, J., and Li, J. (2019, January 8–11). Exploiting Multi-domain Visual Information for Fake News Detection. Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM), Beijing, China.10.1109/ICDM.2019.00062 ↗
18. Wu, K., Yang, S., and Zhu, K.Q. (2015, January 13–17). False rumors detection on Sina Weibo by propagation structures. Proceedings of the 2015 IEEE 31st International Conference on Data Engineering, Seoul, Korea.10.1109/ICDE.2015.7113322 ↗
19. Liu, Y., and Wu, Y.-F.B. (2018, January 2–7). Early Detection of Fake News on Social Media Through Propagation Path Classification with Recurrent and Convolutional Networks. Proceedings of the 32th AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, LA, USA.10.1609/aaai.v32i1.11268 ↗
20. Bidirectional recurrent neural networks10.1109/78.650093 ↗
21. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
22. Kim, Y. (2014, January 25–29). Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar.10.3115/v1/D14-1181 ↗
23. Zhang, Y., and Wallace, B. (2015). A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification. arXiv.
24. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
25. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv.
26. Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhutdinov, R., and Le, Q.V. (2019, January 8–14). XLNet: Generalized Autoregressive Pretraining for Language Understanding. Proceedings of the 32nd Annual Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada.
27. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv.
28. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–23). Non-local Neural Networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
29. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). End-to-End Object Detection with Transformers, Springer.10.1007/978-3-030-58452-8_13 ↗
30. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.B. (2021). Masked Autoencoders Are Scalable Vision Learners. arXiv.10.1109/CVPR52688.2022.01553 ↗
31. Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
32. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013, January 5–10). Distributed Representations of Words and Phrases and their Compositionality. Proceedings of the Advances in Neural Information Processing Systems, Harrahs and Harveys, Lake Tahoe, NV, USA.
33. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). Glove: Global Vectors for Word Representation. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Doha, Qatar.10.3115/v1/D14-1162 ↗
34. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2016). Bag of Tricks for Efficient Text Classification. arXiv.10.18653/v1/E17-2068 ↗
35. Kingma, D.P., and Ba, J. (2015). Adam: A Method for Stochastic Optimization. arXiv.
36. Cui, Y., Che, W., Liu, T., Qin, B., Yang, Z., Wang, S., and Hu, G. (2019). Pre-Training with Whole Word Masking for Chinese BERT. arXiv.
37. Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., and Hu, G. (2020). Revisiting Pre-Trained Models for Chinese Natural Language Processing. arXiv.10.18653/v1/2020.findings-emnlp.58 ↗
38. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., and Parikh, D. (2015, January 7–13). VQA: Visual Question Answering. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.279 ↗
39. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
40. Viualizing data using t-SNE