Extraction of Joint Entity and Relationships with Soft Pruning and GlobalPointer
Source abstract
Abstract
In recent years, scholars have paid increasing attention to the joint entity and relation extraction. However, the most difficult aspect of joint extraction is extracting overlapping triples. To address this problem, we propose a joint extraction model based on Soft Pruning and GlobalPointer, short for SGNet. In the first place, the BERT pretraining model is used to obtain the text word vector representation with contextual information, and then the local and non-local information of the word vector is obtained through graph operations. Specifically, to address the lack of information caused by the rule-based pruning strategies, we utilize the Gaussian Graph Generator and the attention-guiding layer to construct a fully connected graph. This process is called soft pruning for short. Then, to achieve node message passing and information integration, we employ GCNs and a thick connection layer. Next, we use the GlobalPointer decoder to convert triple extraction into quintuple extraction to tackle the problem of problematic overlapping triples extraction. The GlobalPointer decoder, unlike the typical feedforward neural network (FNN), can perform joint decoding. In the end, to evaluate the model performance, the experiment was carried out on two public datasets: the NYT and WebNLG. The experiments show that SGNet performs substantially better on overlapping extraction and achieves good results on two publicly available datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Big Data & Information Engineering, Guizhou University, Guiyang 550025, China
Guizhou Provincial Key Laboratory of Public Big Data, Guizhou University, Guiyang 550025, China
Department of Information and Electronics, Science and Technology College of NCHU, Nanchang 332020, China
Funding
National Natural Science Foundation of China · Guizhou Provincial Science and Technology Projects
References (50)
Show source-provided references
1. Knowledge Graph Construction Techniques
2. Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., and Ives, Z. (2007, January 11–15). DBpedia: A nucleus for a Web of open data. Proceedings of the 6th International Semantic Web Conference, ISWC 2007 and 2nd Asian Semantic Web Conference, ASWC 2007, Busan, Korea.
3. Bollacker, K., Evans, C., Paritosh, P., Sturge, T., and Taylor, J. (2008, January 9–12). Freebase: A collaboratively created graph database for structuring human knowledge. Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data 2008, SIGMOD’08, Vancouver, BC, Canada.10.1145/1376616.1376746 ↗
4. Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E.R., and Mitchell, T.M. (2010, January 11–15). Toward an architecture for never-ending language learning. Proceedings of the AAAI-10/IAAI-10-Proceedings of the 24th AAAI Conference on Artificial Intelligence and the 22nd Innovative Applications of Artificial Intelligence Conference, Atlanta, GA, USA. Available online: https://ojs.aaai.org/index.php/AAAI/article/view/7519.10.1609/aaai.v24i1.7519 ↗
5. Wu, W., Li, H., Wang, H., and Zhu, K.Q. (2012, January 21–24). Probase: A probabilistic taxonomy for text understanding. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD ‘12, Scottsdale, AZ, USA.10.1145/2213836.2213891 ↗
6. Wikidata: A free collaborative knowledgebase10.1145/2629489 ↗
7. Han, X., Liu, Z., and Sun, M. (2018, January 2–7). Neural knowledge acquisition via mutual attention between knowledge graph and text. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, New Orleans, LA, USA. Available online: https://ojs.aaai.org/index.php/AAAI/article/view/11927.10.1609/aaai.v32i1.11927 ↗
8. Luan, Y., He, L., Ostendorf, M., and Hajishirzi, H. (November, January 31). Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1360 ↗
9. Qian, L., Zhou, G., Kong, F., Zhu, Q., and Qian, P. (2008, January 18–22). Exploiting constituent dependencies for tree kernel-based semantic relation extraction. Proceedings of the 22nd International Conference on Computational Linguistics, Manchester, UK.10.3115/1599081.1599169 ↗
10. Kernel Methods for Relation Extraction
11. A Survey of Named Entity Recognition and Classification10.1075/li.30.1.03nad ↗
12. A Survey on Deep Learning for Named Entity Recognition10.1109/TKDE.2020.2981314 ↗
13. Exploring semi-supervised variational autoencoders for biomedical relation extraction10.1016/j.ymeth.2019.02.021 ↗
14. Zhang, Y., Qi, P., and Manning, C.D. (November, January 31). Graph Convolution over Pruned Dependency Trees Improves Relation Extraction. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1244 ↗
15. Dai, D., Xiao, X.Y., Lyu, Y.J., Dou, S., She, Q.Q., and Wang, H.F. (February, January 27). Joint Extraction of Entities and Overlapping Relations Using Position-Attentive Sequence Labeling. Proceedings of the 33rd AAAI Conference on Artificial Intelligence/31st Innovative Applications of Artificial Intelligence Conference/9th AAAI Symposium on Educational Advances in Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.33016300 ↗
16. Ren, X., Wu, Z., He, W., Qu, M., Voss, C.R., Ji, H., Abdelzaher, T.F., and Han, J. (2017, January 3–7). CoType: Joint Extraction of Typed Entities and Relations with Knowledge Bases. Proceedings of the 26th International Conference on World Wide Web (WWW), Perth, Australia.10.1145/3038912.3052708 ↗
17. Li, Q., and Ji, H. (2014, January 22–27). Incremental Joint Extraction of Entity Mentions and Relations. Proceedings of the 52nd Annual Meeting of the Association-for-Computational-Linguistics (ACL), Baltimore, MD, USA.10.3115/v1/P14-1038 ↗
18. Yu, X., and Lam, W. (2010, January 23–27). Jointly identifying entities and extracting relations in encyclopedia text via a graphical model approach. Proceedings of the 23rd International Conference on Computational Linguistics: Posters, Beijing, China.
19. Miwa, M., and Sasaki, Y. (2014, January 25–29). Modeling Joint Entity and Relation Extraction with Table Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1200 ↗
20. Joint entity and relation extraction based on a hybrid neural network10.1016/j.neucom.2016.12.075 ↗
21. Gupta, P., Schütze, H., and Andrassy, B. (2016, January 11–16). Table Filling Multi-Task Recurrent Neural Network for Joint Entity and Relation Extraction. Proceedings of the COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan. Available online: https://aclanthology.org/C16-1239.
22. Tan, Z., Zhao, X., Wang, W., and Xiao, W.D. (February, January 27). Jointly Extracting Multiple Triplets with Multilayer Translation Constraints. Proceedings of the 33rd AAAI Conference on Artificial Intelligence/31st Innovative Applications of Artificial Intelligence Conference/9th AAAI Symposium on Educational Advances in Artificial Intelligence, Honolulu, HI, USA.
23. Zeng, X., He, S., Zeng, D., Liu, K., and Zhao, J. (2019, January 3–7). Learning the extraction order of multiple relational facts in a sentence with reinforcement learning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1035 ↗
24. Fu, T.J., Li, P.H., and Ma, W.Y. (August, January 28). GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction. Proceedings of the 57th Annual Meeting of the Association-for-Computational-Linguistics (ACL), Florence, Italy.
25. Kipf, T.N., and Welling, M. (2017, January 24–26). Semi-supervised classification with graph convolutional networks. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
26. Wei, Z.P., Su, J.L., Wang, Y., Tian, Y., Chang, Y., and Assoc Computat, L. (2020, January 5–10). A Novel Cascade Binary Tagging Framework for Relational Triple Extraction. Proceedings of the 58th Annual Meeting of the Association-for-Computational-Linguistics (ACL), Electr Network.10.18653/v1/2020.acl-main.136 ↗
27. Chan, Y.S., and Roth, D. (2011, January 19–24). Exploiting syntactico-semantic structures for relation extraction. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA. Available online: https://aclanthology.org/P11-1056.
28. GuoDong, Z., Jian, S., Jie, Z., and Min, Z. (2005, January 25–30). Exploring various knowledge in relation extraction. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, Ann Arbor, MI, USA.10.3115/1219840.1219893 ↗
29. Wang, S., Zhang, Y., Che, W., and Liu, T. (2018, January 13–19). Joint extraction of entities and relations based on a novel graph scheme. Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden.10.24963/ijcai.2018/620 ↗
30. Miwa, M., and Bansal, M. (2016, January 7–12). End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures. Proceedings of the 54th Annual Meeting of the Association-for-Computational-Linguistics (ACL), Berlin, Germany.10.18653/v1/P16-1105 ↗
31. Katiyar, A., and Cardie, C. (August, January 30). Going out on a limb: Joint Extraction of Entity Mentions and Relations without Dependency Trees. Proceedings of the 55th Annual Meeting of the Association-for-Computational-Linguistics (ACL), Vancouver, BC, Canada.
32. Zeng, X.R., Zeng, D.J., He, S.Z., Liu, K., and Zhao, J. (2018, January 15–20). Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism. Proceedings of the 56th Annual Meeting of the Association-for-Computational-Linguistics (ACL), Melbourne, Australia.10.18653/v1/P18-1047 ↗
33. Katiyar, A., and Cardie, C. (2016, January 7–12). Investigating LSTMs for joint extraction of opinion entities and relations. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany.10.18653/v1/P16-1087 ↗
34. Zhang, M., Zhang, Y., and Fu, G. (2017, January 9–11). End-to-end neural relation extraction with global optimization. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark.10.18653/v1/D17-1182 ↗
35. Zheng, S.C., Wang, F., Bao, H.Y., Hao, Y.X., Zhou, P., and Xu, B. (August, January 30). Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. Proceedings of the 55th Annual Meeting of the Association-for-Computational-Linguistics (ACL), Vancouver, BC, Canada.
36. Yu, B.W., Zhang, Z.Y., Shu, X.B., Liu, T.W., Wang, Y.B., Wang, B., and Li, S.J. (September, January 29). Joint Extraction of Entities and Relations Based on a Novel Decomposition Strategy. Proceedings of the 24th European Conference on Artificial Intelligence (ECAI), European Assoc Artificial Intelligence, Electr Network.
37. Wang, Y., Yu, B., Zhang, Y., Liu, T., Zhu, H., and Sun, L. (2020, January 13–18). TPLinker: Single-stage Joint Extraction of Entities and Relations through Token Pair Linking. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online).10.18653/v1/2020.coling-main.138 ↗
38. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA.
39. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019, Minneapolis, MN, USA.
40. Xue, F.Z., Sun, A.X., Zhang, H., and Chng, E.S. (2021, January 2–9). GDPNet: Refining Latent Multi-View Graph for Relation Extraction. Proceedings of the 35th AAAI Conference on Artificial Intelligence/33rd Conference on Innovative Applications of Artificial Intelligence/11th Symposium on Educational Advances in Artificial Intelligence, Electr Network.10.1609/aaai.v35i16.17670 ↗
41. Toward scalable internet traffic measurement and analysis with Hadoop10.1145/2427036.2427038 ↗
42. Guo, Z.J., Zhang, Y., and Lu, W. (August, January 28). Attention Guided Graph Convolutional Networks for Relation Extraction. Proceedings of the 57th Annual Meeting of the Association-for-Computational-Linguistics (ACL), Florence, Italy.
43. Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely Connected Convolutional Networks. Proceedings of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
44. Su, J.L. (2021, May 01). GlobalPointer: Deal with Nested and Non-Nested NER in a Unified Way. Available online: https://spaces.ac.cn/archives/8373.
45. Riedel, S., Yao, L., and McCallum, A. (2010). Modeling relations and their mentions without labeled text. Machine Learning and Knowledge Discovery in Databases, Proceedings of the European Conference, ECML PKDD 2010, Barcelona, Spain, 19–23 September 2010, Springer.10.1007/978-3-642-15939-8_10 ↗
46. Gardent, C., Shimorina, A., Narayan, S., and Perez-Beltrachini, L. (August, January 30). Creating Training Corpora for NLG Micro-Planning. Proceedings of the 55th Annual Meeting of the Association-for-Computational-Linguistics (ACL), Vancouver, BC, Canada.
47. Loshchilov, I., and Hutter, F. (2019). Decoupled weight decay regularization. arXiv.
48. Nayak, T., and Ng, H.T. (2020, January 7–12). Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction. Proceedings of the 34th AAAI Conference on Artificial Intelligence/32nd Innovative Applications of Artificial Intelligence Conference/10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6374 ↗
49. Ma, L., Ren, H., and Zhang, X. (2021). Effective Cascade Dual-Decoder Model for Joint Entity and Relation Extraction. arXiv.
50. RMAN: Relational multi-head attention neural network for joint extraction of entities and relations10.1007/s10489-021-02600-2 ↗