Knowledge Graph Completion Using a Pre-Trained Language Model Based on Categorical Information and Multi-Layer Residual Attention
Source abstract
Abstract
Knowledge graph completion (KGC) utilizes known knowledge graph triples to infer and predict missing knowledge, making it one of the research hotspots in the field of knowledge graphs. There are still limitations in generating high-quality entity embeddings and fully understanding the contextual information of entities and relationships. To overcome these challenges, this paper introduces a novel pre-trained language model-based method for knowledge graph completion that significantly enhances the quality of entity embeddings by integrating entity categorical information with textual descriptions. Additionally, this method employs an innovative multi-layer residual attention network in combination with PLMs, deepening the understanding of the joint contextual information of entities and relationships. Experimental results on the FB15k-237 and WN18RR datasets demonstrate that our proposed model significantly outperforms existing baseline models in link prediction tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Graph Neural Networks · Topic Modeling · Data Quality and Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of China’s Ethnic Languages and Information Technology of Ministry of Education, Northwest Minzu University, Lanzhou 730030, China
School of Mathematics and Computer Science, Northwest Minzu Univsersity, Lanzhou 730030, China
Funding
2023 Basic Research Operating Expenses of Central Universities · National Natural Science Foundation of China · Natural Science Foundation of Gansu Province
References (22)
Show source-provided references
1. Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., and Yakhnenko, O. (2013). Advances in Neural Information Processing Systems 26, Proceedings of the 27th Annual Conference on Neural Information Processing Systems 2013, Lake Tahoe, NV, USA, 5–10 December 2013, Curran Associates, Inc.
2. Sun, Z., Deng, Z.H., Nie, J.Y., and Tang, J. (2019). Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv.
3. Yao, L., Mao, C., and Luo, Y. (2019). KG-BERT: BERT for knowledge graph completion. arXiv.
4. Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion
5. Wang, X., He, Q., Liang, J., and Xiao, Y. (2022, January 23–29). Language Models as Knowledge Embeddings. Proceedings of the International Joint Conference on Artificial Intelligence, Vienna, Austria.10.24963/ijcai.2022/318 ↗
6. Wang, Z., Zhang, J., Feng, J., and Chen, Z. (2014, January 27–31). Knowledge Graph Embedding by Translating on Hyperplanes. Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, Québec, QC, Canada.10.1609/aaai.v28i1.8870 ↗
7. Lin, Y., Liu, Z., Sun, M., Liu, Y., and Zhu, X. (2015, January 25–30). Learning Entity and Relation Embeddings for Knowledge Graph Completion. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9491 ↗
8. Ji, G., Liu, K., He, S., and Zhao, J. (2016, January 12–17). Knowledge Graph Completion with Adaptive Sparse Transfer Matrix. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10089 ↗
9. Chao, L., He, J., Wang, T., and Chu, W. (2020). PairRE: Knowledge graph embeddings via paired relation vectors. arXiv.10.18653/v1/2021.acl-long.336 ↗
10. Nickel, M., Volker, T., and Hans-Peter, K. (July, January 28). A Three-Way Model for Collective Learning on Multi-Relational Data. Proceedings of the 28th International Conference on Machine Learning, Bellevue, DC, USA.
11. Yang, B., Yih, W., He, X., Gao, J., and Deng, L. (2014). Embedding Entities and Relations for Learning and Inference in Knowledge Bases. arXiv.
12. Théo, T., Johannes, W., Sebastian, R., Éric, G., and Guillaume, B. (2016, January 19–24). Complex Embeddings for Simple Link Prediction. Proceedings of the 33rd International Conference on International Conference on Machine Learning, New York, NY, USA.
13. Nickel, M., Rosasco, L., and Poggio, T. (2016, January 12–17). Holographic Embeddings of Knowledge Graphs. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10314 ↗
14. Kazemi, S., and Poole, N. (2018). Advances in Neural Information Processing Systems 31, Proceedings of the Annual Conference on Neural Information Processing Systems 2018, Montréal, QC, Canada, 3–8 December 2018, Curran Associates, Inc.
15. Bilinear Scoring Function Search for Knowledge Graph Learning10.1109/TPAMI.2022.3157321 ↗
16. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Advances in Neural Information Processing Systems, Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017, Curran Associates, Inc.
17. Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019, January 6–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of naacL-HLT, Minneapolis, MN, USA.
18. Enhancing the Detection of Misogynistic Content in Social Media by Transferring Knowledge from Song Phrases10.1109/ACCESS.2023.3242965 ↗
19. Zhang, Z., Liu, X., Zhang, Y., Su, Q., Sun, X., and He, B. (2020). Findings of the Association for Computational Linguistics, Association for Computational Linguistics.
20. MEM-KGC: Masked entity model for knowledge graph completion with pre-trained language model10.1109/ACCESS.2021.3113329 ↗
21. Wang, L., Zhao, W., Wei, Z., and Liu, J. (2022, January 22–27). SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.10.18653/v1/2022.acl-long.295 ↗
22. Iulia Turc, T. (2019). Well-read students learn better: On the importance of pre-training compact models. arXiv.