Syntax-Informed Self-Attention Network for Span-Based Joint Entity and Relation Extraction
Source abstract
Abstract
Current state-of-the-art joint entity and relation extraction framework is based on span-level entity classification and relation identification between pairs of entity mentions. However, while maintaining an efficient exhaustive search on spans, the importance of syntactic features is not taken into consideration. It will lead to a problem that the prediction of a relation between two entities is related based on corresponding entity types, but in fact they are not related in the sentence. In addition, although previous works have proven that extract local context is beneficial for the task, it still lacks in-depth learning of contextual features in local context. In this paper, we propose to incorporate syntax knowledge into multi-head self-attention by employing part of heads to focus on syntactic parents of each token from pruned dependency trees, and we use it to model the global context to fuse syntactic and semantic features. In addition, in order to get richer contextual features from the local context, we apply local focus mechanism on entity pairs and corresponding context. Based on applying the two strategies, we perform joint entity and relation extraction on span-level. Experimental results show that our model achieves significant improvements on both Conll04 and SciERC dataset compared to strong competitors.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Biomedical Text Mining and Ontologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
Funding
National Natural Science Foundation of China
References (58)
Show source-provided references
1. Yu, M., Yin, W., Hasan, K.S., dos Santos, C., Xiang, B., and Zhou, B. (August, January 30). Improved Neural Relation Detection for Knowledge Base Question Answering. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
2. Zhang, Y., Zhong, V., Chen, D., Angeli, G., and Manning, C.D. (2017, January 7–11). Position-aware attention and supervised data improve slot filling. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1004 ↗
3. Zeng, D., Liu, K., Lai, S., Zhou, G., and Zhao, J. (2014, January 25–29). Relation Classification via Convolutional Deep Neural Network. Proceedings of the COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland.
4. Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., and Xu, B. (2016, January 7–12). Attention-based bidirectional long short-term memory networks for relation classification. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Berlin, Germany.10.18653/v1/P16-2034 ↗
5. Wang, L., Cao, Z., De Melo, G., and Liu, Z. (2016, January 7–12). Relation classification via multi-level attention cnns. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1123 ↗
6. Miwa, M., and Bansal, M. (2016, January 7–12). End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1105 ↗
7. Katiyar, A., and Cardie, C. (August, January 30). Going out on a limb: Joint extraction of entity mentions and relations without dependency trees. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
8. Zheng, S., Wang, F., Bao, H., Hao, Y., Zhou, P., and Xu, B. (August, January 30). Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
9. Zeng, X., Zeng, D., He, S., Liu, K., and Zhao, J. (2018, January 15–20). Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.10.18653/v1/P18-1047 ↗
10. Joint entity recognition and relation extraction as a multi-head selection problem10.1016/j.eswa.2018.07.032 ↗
11. Fu, T.J., Li, P.H., and Ma, W.Y. (August, January 28). GraphRel: Modeling text as relational graphs for joint entity and relation extraction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
12. Dixit, K., and Al-Onaizan, Y. (August, January 28). Span-Level Model for Relation Extraction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
13. Eberts, M., and Ulges, A. (September, January 29). Span-based Joint Entity and Relation Extraction with Transformer Pre-training. Proceedings of the 24th European Conference on Artificial Intelligence, Santiago de Compostela, Spain.
14. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (December, January 4). Attention is all you need. Proceedings of the 30th Advances in Neural Information Processing Systems, Long Beach, CA, USA.
15. Li, J., Tu, Z., Yang, B., Lyu, M.R., and Zhang, T. (November, January 31). Multi-Head Attention with Disagreement Regularization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1317 ↗
16. Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. (August, January 28). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
17. Michel, P., Levy, O., and Neubig, G. (December, January 8). Are Sixteen Heads Really Better than One?. Proceedings of the 32th Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
18. Vig, J. (August, January 28). A Multiscale Visualization of Attention in the Transformer Model. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Florence, Italy.
19. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
20. Du, C., Sun, H., Wang, J., Qi, Q., and Liao, J. (2020, January 5–10). Adversarial and domain-aware bert for cross-domain sentiment analysis. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.370 ↗
21. Yamada, I., Asai, A., Shindo, H., Takeda, H., and Matsumoto, Y. (2020). LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention. arXiv.10.18653/v1/2020.emnlp-main.523 ↗
22. He, P., Liu, X., Gao, J., and Chen, W. (2020). DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv.
23. Kernel methods for relation extraction
24. Bunescu, R.C., and Mooney, R.J. (2005, January 6–8). A shortest path dependency kernel for relation extraction. Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Vancouver, BC, Canada.10.3115/1220575.1220666 ↗
25. Xu, K., Feng, Y., Huang, S., and Zhao, D. (2015, January 17–21). Semantic Relation Classification via Convolutional Neural Networks with Simple Negative Sampling. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1062 ↗
26. Xu, Y., Mou, L., Li, G., Chen, Y., Peng, H., and Jin, Z. (2015, January 17–21). Classifying relations via long short term memory networks along shortest dependency paths. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1206 ↗
27. Cai, R., Zhang, X., and Wang, H. (2016, January 7–12). Bidirectional recurrent convolutional neural network for relation classification. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1072 ↗
28. He, Z., Chen, W., Li, Z., Zhang, M., Zhang, W., and Zhang, M. (2018, January 2–7). SEE: Syntax-aware entity embedding for neural relation extraction. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12042 ↗
29. Tai, K.S., Socher, R., and Manning, C.D. (2015, January 26–31). Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China.10.3115/v1/P15-1150 ↗
30. Kipf, T.N., and Welling, M. (2017, January 24–26). Semi-Supervised Classification with Graph Convolutional Networks. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France.
31. Zhang, Y., Qi, P., and Manning, C.D. (November, January 31). Graph Convolution over Pruned Dependency Trees Improves Relation Extraction. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1244 ↗
32. Veyseh, A.P.B., Dernoncourt, F., Dou, D., and Nguyen, T.H. (2020, January 5–10). Exploiting the Syntax-Model Consistency for Neural Relation Extraction. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.
33. Shen, Y., Tan, S., Sordoni, A., and Courville, A. (2019, January 6–9). Ordered neurons: Integrating tree structures into recurrent neural networks. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA.
34. Shaw, P., Uszkoreit, J., and Vaswani, A. (2018, January 1–6). Self-Attention with Relative Position Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, LA, USA.10.18653/v1/N18-2074 ↗
35. Wang, X., Tu, Z., Wang, L., and Shi, S. (2019, January 3–7). Self-Attention with Structural Position Representations. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1145 ↗
36. Miwa, M., and Sasaki, Y. (2014, January 25–29). Modeling joint entity and relation extraction with table representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1200 ↗
37. Li, Q., and Ji, H. (2014, January 22–27). Incremental joint extraction of entity mentions and relations. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, MD, USA.10.3115/v1/P14-1038 ↗
38. Ren, X., Wu, Z., He, W., Qu, M., Voss, C.R., Ji, H., Abdelzaher, T.F., and Han, J. (2017, January 3–7). Cotype: Joint extraction of typed entities and relations with knowledge bases. Proceedings of the 26th International Conference on World Wide Web, Perth, Australia.10.1145/3038912.3052708 ↗
39. Joint entity and relation extraction based on a hybrid neural network10.1016/j.neucom.2016.12.075 ↗
40. Gupta, P., Schütze, H., and Andrassy, B. (2016, January 11–16). Table filling multi-task recurrent neural network for joint entity and relation extraction. Proceedings of the COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan.
41. Zhang, M., Zhang, Y., and Fu, G. (2017, January 7–11). End-to-end neural relation extraction with global optimization. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1182 ↗
42. Chi, R., Wu, B., Hu, L., and Zhang, Y. (2019, January 1–3). Enhancing Joint Entity and Relation Extraction with Language Modeling and Hierarchical Attention. Proceedings of the Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data, Chengdu, China.10.1007/978-3-030-26072-9_24 ↗
43. Nguyen, D.Q., and Verspoor, K. (2019, January 14–18). End-to-end neural relation extraction using deep biaffine attention. Proceedings of the European Conference on Information Retrieval, Cologne, Germany.10.1007/978-3-030-15712-8_47 ↗
44. Wang, Y., Yu, B., Zhang, Y., Liu, T., Zhu, H., and Sun, L. (2020, January 13–18). TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.138 ↗
45. Luan, Y., Wadden, D., He, L., Shah, A., Ostendorf, M., and Hajishirzi, H. (2019, January 2–7). A general framework for information extraction using dynamic span graphs. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.10.18653/v1/N19-1308 ↗
46. Wadden, D., Wennberg, U., Luan, Y., and Hajishirzi, H. (2019, January 3–7). Entity, Relation, and Event Extraction with Contextualized Span Representations. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1585 ↗
47. Li, X., Yin, F., Sun, Z., Li, X., Yuan, A., Chai, D., Zhou, M., and Li, J. (August, January 28). Entity-Relation Extraction as Multi-Turn Question Answering. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
48. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv.
49. Liu, X., Duh, K., Liu, L., and Gao, J. (2020). Very deep transformers for neural machine translation. arXiv.
50. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv.
51. Roth, D., and Yih, W.t. (2004, January 6–7). A Linear Programming Formulation for Global Inference in Natural Language Tasks. Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, Boston, MA, USA.
52. Luan, Y., He, L., Ostendorf, M., and Hajishirzi, H. (November, January 31). Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1360 ↗
53. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2019). HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv.10.18653/v1/2020.emnlp-demos.6 ↗
54. Beltagy, I., Lo, K., and Cohan, A. (2019, January 3–7). SciBERT: A pretrained language model for scientific text. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1371 ↗
55. Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J.R., Bethard, S., and McClosky, D. (2014, January 22–27). The Stanford CoreNLP natural language processing toolkit. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Baltimore, MD, USA.10.3115/v1/P14-5010 ↗
56. Yao, Y., Ye, D., Li, P., Han, X., Lin, Y., Liu, Z., Liu, Z., Huang, L., Zhou, J., and Sun, M. (August, January 28). DocRED: A Large-Scale Document-Level Relation Extraction Dataset. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
57. Nan, G., Guo, Z., Sekulic, I., and Lu, W. (2020, January 5–10). Reasoning with Latent Structure Refinement for Document-Level Relation Extraction. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.141 ↗
58. Zeng, S., Xu, R., Chang, B., and Li, L. (2020). Double Graph Based Reasoning for Document-level Relation Extraction. arXiv.10.18653/v1/2020.emnlp-main.127 ↗