journal article·2025-09-12Open access

Dual-Stream Transformer with LLM-Empowered Symbol Drift Modeling for Health Misinformation Detection

Jingsheng Wang, Zhengjie Fu, Chenlu Jiang, Manzhou Li, Yan Zhan

Applied Sciences· Volume 15· Issue 18

Abstract

In the era of big-data-driven multi-platform and multimodal health information dissemination, the rapid spread of false and misleading content poses a critical threat to public health awareness and decision making. To address this issue, a dual-stream Transformer-based multimodal health misinformation detection framework is presented, incorporating a symbol drift detection module, a symbol-aware text graph neural network, and a crossmodal alignment fusion module. The framework enables precise identification of implicit misleading health-related symbols, comprehensive modeling of textual dependency structures, and robust detection of crossmodal semantic conflicts. A domain-specific health-symbol-sensitive lexicon is constructed, and contextual drift intensity is quantitatively measured and embedded as explicit features into the text GNN. Bidirectional cross-attention and contrastive learning are further employed to enhance crossmodal semantic alignment. Extensive experiments on a large-scale real-world multimodal health information dataset, encompassing heterogeneous data sources typical of big data environments, demonstrate that the proposed method consistently outperforms state-of-the-art baselines in CTR prediction, multimodal recommendation, and ranking tasks. The results indicate substantial improvements in both accuracy and ranking quality, while ablation studies further verify the contributions of symbol drift modeling, graph-structured representation, and crossmodal fusion. Overall, the proposed approach advances big data analytics for multimodal misinformation detection and provides an interpretable and scalable solution for public health communication governance.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Misinformation and Its Impacts · Topic Modeling · Text and Document Classification Technologies

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of Journalism and Communication, Northwest University, Xi’an 710127, China

National School of Development, Peking University, Beijing 100871, China

China Agricultural University, Beijing 100083, China

Artificial Intelligence Research Institute, Tsinghua University, Beijing 100084, China

Funding

National Natural Science Foundation of China

References (46)

Show source-provided references
1. Chen, A., Wei, Y., Le, H., and Zhang, Y. (Br. J. Educ. Technol., 2024). Learning by teaching with ChatGPT: The effect of teachable ChatGPT agent on programming education, Br. J. Educ. Technol., early view.10.1111/bjet.70001 ↗
2. Chen, J., Sun, B., Li, H., Lu, H., and Hua, X.S. (2016, January 15–19). Deep ctr prediction in display advertising. Proceedings of the 24th ACM international conference on Multimedia, Amsterdam, The Netherlands.10.1145/2964284.2964325 ↗
3. Zhao, W., Zhang, J., Xie, D., Qian, Y., Jia, R., and Li, P. (2019, January 3–7). AIBox: CTR prediction model training on a single node. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.10.1145/3357384.3358045 ↗
4. Wang, Q., Liu, F., Zhao, X., and Tan, Q. (2022). A CTR prediction model based on session interest. PLoS ONE, 17.10.1371/journal.pone.0273048 ↗
5. Dynamic user modeling in social media systems10.1145/2699670 ↗
6. Zhou, C., Bai, J., Song, J., Liu, X., Zhao, Z., Chen, X., and Gao, J. (2018, January 2–7). Atrank: An attention-based user behavior modeling framework for recommendation. Proceedings of the AAAI conference on artificial intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11618 ↗
7. Lan, L., and Geng, Y. (February, January 27). Accurate and interpretable factorization machines. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
8. Wu, S., Li, Z., Su, Y., Cui, Z., Zhang, X., and Wang, L. (2021). GraphFM: Graph factorization machines for feature interaction modeling. arXiv.
9. Yi, Y., Zhou, Y., Wang, T., and Zhou, J. (2025). Advances in Video Emotion Recognition: Challenges and Trends. Sensors, 25.10.3390/s25123615 ↗
10. Multimodal fusion for audio-image and video action recognition10.1007/s00521-023-09186-5 ↗
11. Kraprayoon, J., Pham, A., and Tsai, T.J. (2024). Improving the robustness of DTW to global time warping conditions in audio synchronization. Appl. Sci., 14.10.3390/app14041459 ↗
12. Zhang, Y., Wu, J., Sun, L., and Yang, G. (2025). Contrastive Learning-Based Cross-Modal Fusion for Product Form Imagery Recognition: A Case Study on New Energy Vehicle Front-End Design. Sustainability, 17.10.3390/su17104432 ↗
13. Zhang, L., Zhang, Y., and Ma, X. (2021, January 17–19). A new strategy for tuning ReLUs: Self-adaptive linear units (SALUs). Proceedings of the ICMLCA 2021; 2nd International Conference on Machine Learning and Computer Application, Shenyang, China.
14. A comprehensive survey on advertising click-through rate prediction algorithm10.1017/S0269888925000025 ↗
15. Zhou, X., Chen, S., Ren, Y., Zhang, Y., Fu, J., Fan, D., Lin, J., and Wang, Q. (2022). Atrous Pyramid GAN Segmentation Network for Fish Images with High Performance. Electronics, 11.10.3390/electronics11060911 ↗
16. Click-through rate prediction model integrating user interest and multi-head attention mechanism10.1186/s40537-023-00688-6 ↗
17. He, L., Chen, H., Wang, D., Jameel, S., Yu, P., and Xu, G. (2021, January 1–5). Click-through rate prediction with multi-modal hypergraphs. Proceedings of the 30th ACM International Conference on Information & Knowledge Management, Virtual Event, QLD, Australia.10.1145/3459637.3482327 ↗
18. Deng, J., Shen, D., Wang, S., Wu, X., Yang, F., Zhou, G., and Meng, G. (2023). ContentCTR: Frame-level live streaming click-through rate prediction with multimodal transformer. arXiv.
19. Deng, K., and Woodland, P.C. (2025). Multi-head Temporal Latent Attention. arXiv.
20. Blondel, M., Fujino, A., Ueda, N., and Ishihata, M. (2016, January 5–10). Higher-order factorization machines. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain.
21. Guo, H., Tang, R., Ye, Y., Li, Z., and He, X. (2017). DeepFM: A factorization-machine based neural network for CTR prediction. arXiv.10.24963/ijcai.2017/239 ↗
22. Zhou, G., Zhu, X., Song, C., Fan, Y., Zhu, H., Ma, X., Yan, Y., Jin, J., Li, H., and Gai, K. (2018, January 19–23). Deep interest network for click-through rate prediction. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219823 ↗
23. de Souza Pereira Moreira, G., Rabhi, S., Lee, J.M., Ak, R., and Oldridge, E. (October, January 27). Transformers4rec: Bridging the gap between nlp and sequential/session-based recommendation. Proceedings of the 15th ACM Conference on Recommender Systems, Amsterdam, The Netherlands.
24. Yang, Y., Zhang, L., and Liu, J. (2025). Temporal user interest modeling for online advertising using Bi-LSTM network improved by an updated version of Parrot Optimizer. Sci. Rep., 15.10.1038/s41598-025-03208-z ↗
25. Wei, Y., Wang, X., Nie, L., He, X., Hong, R., and Chua, T.S. (2019, January 21–25). MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3351034 ↗
26. Li, T., Yang, X., Ke, Y., Wang, B., Liu, Y., and Xu, J. (2024, January 13–16). Alleviating the inconsistency of multimodal data in cross-modal retrieval. Proceedings of the 2024 IEEE 40th International Conference on Data Engineering (ICDE), Utrecht, The Netherlands.10.1109/ICDE60146.2024.00353 ↗
27. Kim, W., Son, B., and Kim, I. (2021, January 18–24). Vilt: Vision-and-language transformer without convolution or region supervision. Proceedings of the International Conference on Machine Learning, Virtual.
28. Vision-language pre-training: Basics, recent advances, and future trends
29. Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. (2022, January 18–24). Flava: A foundational language and vision alignment model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01519 ↗
30. Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. (November, January 27). Videobert: A joint model for video and language representation learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
31. Li, M., Xu, R., Wang, S., Zhou, L., Lin, X., Zhu, C., Zeng, M., Ji, H., and Chang, S.F. (2022, January 18–24). Clip-event: Connecting text and images with event structures. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01593 ↗
32. Epidemic model for information diffusion in web forums: Experiments in marketing exchange and political dialog10.1186/s40064-016-1675-x ↗
33. Xu, Z., and Qian, M. (2023). Predicting popularity of viral content in social media through a temporal-spatial cascade convolutional learning framework. Mathematics, 11.10.3390/math11143059 ↗
34. Deep Learning based Optimal Multimodal Fusion Framework for Intrusion Detection Systems for Healthcare Data
35. Song, Y., Elkahky, A.M., and He, X. (2016, January 17–21). Multi-rate deep learning for temporal recommendation. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy.10.1145/2911451.2914726 ↗
36. Dean, S., Dong, E., Jagadeesan, M., and Leqi, L. (2024, January 26–27). Recommender systems as dynamical systems: Interactions with viewers and creators. Proceedings of the Workshop on Recommendation Ecosystems: Modeling, Optimization and Incentive Design, Vancouver, BC, Canada.
37. Sangiorgio, E., Di Marco, N., Etta, G., Cinelli, M., Cerqueti, R., and Quattrociocchi, W. (2025). Evaluating the effect of viral posts on social media engagement. Sci. Rep., 15.10.1038/s41598-024-84960-6 ↗
38. Multimodal attention-based transformer for video captioning10.1007/s10489-023-04597-2 ↗
39. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
40. Cheng, H.T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G., Chai, W., and Ispir, M. (2016, January 15). Wide & deep learning for recommender systems. Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, Boston, MA, USA.10.1145/2988450.2988454 ↗
41. Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., and Chi, E.H. (2018, January 19–23). Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3220007 ↗
42. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning10.1016/j.neucom.2022.07.028 ↗
43. Zhou, X. (2023, January 6–8). Mmrec: Simplifying multimodal recommendation. Proceedings of the 5th ACM International Conference on Multimedia in Asia Workshops, Tainan, Taiwan.10.1145/3611380.3628561 ↗
44. From ranknet to lambdarank to lambdamart: An overview
45. Ultrasound volume projection image quality selection by ranking from convolutional RankNet10.1016/j.compmedimag.2020.101847 ↗
46. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.Y. (2017, January 4–9). Lightgbm: A highly efficient gradient boosting decision tree. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.