journal article·2025-11-07Open access

MPCFN: A Multilevel Predictive Cross-Fusion Network for Multimodal Named Entity Recognition in Social Media

Qinjun Qiu, Bo Tan, Yukuan Zhou, Wenjing Chen, Miao Tian, Liufeng Tao

Applied Sciences· Volume 15· Issue 22

Abstract

The goal of the Multimodal Named Entity Recognition (MNER) job is to identify and classify named entities by combining various data modalities (such as text and images) and assigning them to specified categories. The growing prevalence of multimodal social media posts has spurred heightened interest in MNER, particularly due to its pivotal role in applications ranging from intention comprehension to personalized user recommendations. In the MNER task, the inconsistency between image information and text information and the difficulty of fully utilizing the image information to complement the text information are the two main difficulties currently faced. In order to solve these problems, this study proposes a Multilevel Predictive Cross-Fusion Network (MPCFN) approach for Multimodal Named Entity Recognition. First, textual features are extracted using BERT and visual features are extracted using ResNet, then irrelevant information in the image is filtered using the Correlation Prediction Gate. Second, the hierarchy of visual features received by each Transformer block is controlled by the Dynamic Gate and aligned between image and textual features using the Cross-Fusion Module to align the image and text features. Finally, the hidden layer representation is fed into the CRF layer optimized for decoding using Flooding. Through experiments on TWITTER-2015, TWITTER-2017, and WuKong datasets, our method achieves F1 scores of 76.74%, 87.61%, and 82.35%, outperforming the existing mainstream state-of-the-art models and proving the effectiveness and superiority of our method.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Topic Modeling · Multimodal Machine Learning Applications · Text and Document Classification Technologies

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of Computer Science, China University of Geosciences, Wuhan 430074, China

Yunnan Key Laboratory of Intelligent-Monitoring and Spatiotemporal Big Data Governance of Natural Resources, Kunming 650051, China

Yunnan Institute of Geology and Mineral Surveying and Mapping Co., Ltd., Kunming 650051, China

The Land and Resources Information Center, Department of Natural Resources of Yunnan Province, Kunming 650224, China

Funding

National Key R&D Program of China · Open Fund Program of Yunnan Key Laboratory of Intelligent Monitoring and Spatiotemporal Big Data Governance of Natural Resources

References (44)

Show source-provided references
1. Yu, J., Bohnet, B., and Poesio, M. (2020, January 5–10). Named Entity Recognition as Dependency Parsing. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.577 ↗
2. Named entity recognition and relation extraction: State-of-the-art10.1145/3445965 ↗
3. Kim, J.H., and Woodland, P.C. (2000, January 16–20). A rule-based named entity recognition system for speech input. Proceedings of the Sixth International Conference on Spoken Language Processing, Beijing, China.10.21437/ICSLP.2000-131 ↗
4. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA. long and short papers.
5. Jie, Z., and Lu, W. (2019, January 3–7). Dependency-Guided LSTM-CRF for Named Entity Recognition. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1399 ↗
6. Wu, Z., Zheng, C., Cai, Y., Chen, J., Leung, H., and Li, Q. (2020, January 12–16). Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3413650 ↗
7. Object-aware multimodal named entity recognition in social media posts with adversarial learning10.1109/TMM.2020.3013398 ↗
8. Yu, J., Jiang, J., Yang, L., and Xia, R. (2020, January 5–10). Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. Proceedings of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.306 ↗
9. Lu, D., Neves, L., Carvalho, V., Zhang, N., and Ji, H. (2018, January 15–20). Visual attention model for name tagging in multimodal social media. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia. Long Papers.10.18653/v1/P18-1185 ↗
10. Moon, S., Neves, L., and Carvalho, V. (2018, January 1–6). Multimodal Named Entity Recognition for Short Social Media Posts. Proceedings of the NAACL-HLT, New Orleans, LA, USA.10.18653/v1/N18-1078 ↗
11. HVP-Net: A hybrid voxel-and point-wise network for place recognition10.1109/TIV.2023.3308116 ↗
12. Wang, X., Gui, M., Jiang, Y., Jia, Z., Bach, N., Wang, T., Huang, Z., Huang, F., and Tu, K. (2022, January 10–15). ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA.10.18653/v1/2022.naacl-main.232 ↗
13. Jia, M., Shen, X., Shen, L., Pang, J., Liao, L., Song, Y., Chen, M., and He, X. (2022, January 10–14). Query prior matters: A MRC framework for multimodal named entity recognition. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3548427 ↗
14. Lu, J., Zhang, D., and Zhang, P. (2022, January 12–17). Flat Multi-modal Interaction Transformer for Named Entity Recognition. Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea.
15. CWI: A multimodal deep learning approach for named entity recognition from social media using character, word and image features10.1007/s00521-021-06488-4 ↗
16. Wang, X., Ye, J., Li, Z., Tian, J., Jiang, Y., Yan, J., Zhang, J., and Xiao, Y. (July, January 18). CAT-MNER: Multimodal named entity recognition with knowledge-refined cross-modal attention. Proceedings of the 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan.10.1109/ICME52920.2022.9859972 ↗
17. Xu, B., Huang, S., Sha, C., and Wang, H. (2022, January 21–25). MAF: A general matching and alignment framework for multimodal named entity recognition. Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, Tempe, AZ, USA.10.1145/3488560.3498475 ↗
18. Kapur, J.N. (1989). Maximum-Entropy Models in Science and Engineering, John Wiley & Sons.
19. Hidden markov models10.1016/S0959-440X(96)80056-X ↗
20. Support vector machines10.1109/5254.708428 ↗
21. Lafferty, J., McCallum, A., and Pereira, F. (July, January 28). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Proceedings of the ICML, Williamstown, MA, USA.
22. Borthwick, A., Sterling, J., Agichtein, E., and Grishman, R. (May, January 29). NYU: Description of the MENE named entity system as used in MUC-7. Proceedings of the Seventh Message Understanding Conference (MUC-7), Fairfax, VA, USA.
23. Mccallum, A., and Li, W. (1, January May). Early results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons. Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, Edmonton, AB, Canada.10.3115/1119176.1119206 ↗
24. EPIC: An epidemiological investigation of COVID-19 dataset for Chinese named entity recognition10.1016/j.ipm.2023.103541 ↗
25. Named entity recognition with bidirectional LSTM-CNNs10.1162/tacl_a_00104 ↗
26. Disease named entity recognition from biomedical literature using a novel convolutional neural network10.1186/s12920-017-0316-8 ↗
27. Zheng, S., Wang, F., Bao, H., Hao, Y., Zhou, P., and Xu, B. (August, January 30). Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada. Long Papers.
28. Gui, T., Ye, J., Zhang, Q., Zhou, Y., Gong, Y., and Huang, X. (2021, January 19–27). Leveraging document-level label consistency for named entity recognition. Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, Montreal, QC, Canada.10.24963/ijcai.2020/550 ↗
29. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving language understanding by generative pre-training. Comput. Sci., in press.
30. Robust scientific text classification using prompt tuning based on data augmentation with L2 regularization10.1016/j.ipm.2023.103531 ↗
31. USAF: Multimodal Chinese named entity recognition using synthesized acoustic features10.1016/j.ipm.2023.103290 ↗
32. Zhang, Q., Fu, J., Liu, X., and Huang, X. (2018, January 2–7). Adaptive co-attention network for named entity recognition in tweets. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11962 ↗
33. Chen, D., Li, Z., Gu, B., and Chen, Z. (2021). Multimodal named entity recognition with image attributes and image knowledge. Database Systems for Advanced Applications, Proceedings of the 26th International Conference, DASFAA 2021, Taipei, Taiwan, 11–14 April 2021, Springer International Publishing.10.1007/978-3-030-73197-7_12 ↗
34. Sun, L., Wang, J., Zhang, K., Su, Y., and Weng, F. (2021, January 19–21). RpBERT: A text-image relation propagation-based BERT model for multimodal NER. Proceedings of the AAAI Conference on Artificial Intelligence, Online.10.1609/aaai.v35i15.17633 ↗
35. Multi-granularity cross-modal representation learning for named entity recognition on social media10.1016/j.ipm.2023.103546 ↗
36. Wang, X., Cai, J., Jiang, Y., Xie, P., Tu, K., and Lu, W. (2022, January 7–11). Named Entity and Relation Extraction with Multi-Modal Retrieval. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates.10.18653/v1/2022.findings-emnlp.437 ↗
37. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv.
38. Attention is all you need
39. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016, January 12–17). Neural Architectures for Named Entity Recognition. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1030 ↗
40. Ishida, T., Yamane, I., Sakai, T., Niu, G., and Sugiyama, M. (2020, January 13–18). Do We Need Zero Training Loss After Achieving Zero Training Error?. Proceedings of the International Conference on Machine Learning, PMLR, Online.
41. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark
42. A survey on deep learning for named entity recognition10.1109/TKDE.2020.2981314 ↗
43. Zhang, D., Wei, S., Li, S., Wu, H., Zhu, Q., and Zhou, G. (2021, January 19–21). Multi-modal graph fusion for named entity recognition with targeted visual guidance. Proceedings of the AAAI Conference on Artificial Intelligence, Online.10.1609/aaai.v35i16.17687 ↗
44. Yu, J., Li, Z., Wang, J., and Xia, R. (2023, January 9–14). Grounded multimodal named entity recognition on social media. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, ON, Canada.10.18653/v1/2023.acl-long.508 ↗