FA-RCNet: A Fused Feature Attention Network for Relationship Classification
Source abstract
Abstract
Relation extraction is an important task in natural language processing. It plays an integral role in intelligent question-and-answer systems, semantic search, and knowledge graph work. For this task, previous studies have demonstrated the effectiveness of convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) in relational classification tasks. Recently, due to the superior performance of the pre-trained model BERT, BERT has become a feature extraction module for many relational classification models, and good results have been achieved in work related to BERT. However, most of such work uses the deepest levels of features. The important role of shallow-level information in the relational classification task is ignored. Based on the above problems, a relationship classification network FA-RCNet (fusion-attention relationship classification network) with feature fusion and attention mechanism is proposed in this paper. FA-RCNet fuses shallow-level features with deep-level features, and augments entity features and global features by the attention module so that the feature vector can perform the relational classification task more perfectly. In addition, the model in this paper achieves advanced results on both the SemEval-2010 Task 8 dataset and the KBP37 dataset compared to previously published models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Advanced Graph Neural Networks
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 250316, China
Funding
National Key R&D Plan of China
References (51)
Show source-provided references
1. Pawar, S., Palshikar, G.K., and Bhattacharyya, P. (2017). Relation Extraction: A Survey. arXiv.
2. A distributed meta-learning system for Chinese entity relation extraction10.1016/j.neucom.2014.07.016 ↗
3. A new sentence similarity measure and sentence based extractive technique for automatic text summarization10.1016/j.eswa.2008.11.022 ↗
4. Girju, R. (2003, July 11). Automatic Detection of Causal Relations for Question Answering. Available online: https://aclanthology.org/W03-1210.pdf.
5. Bahdanau, D., Cho, K., and Bengio, Y. (2016). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
6. Knowledge Graph Embedding: A Survey of Approaches and Applications10.1109/TKDE.2017.2754499 ↗
7. Kate, R.J., and Mooney, R.J. (2006, January 17–21). Using string-kernels for learning semantic parsers. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia.10.3115/1220175.1220290 ↗
8. Paramesha, K., and Ravishankar, K.C. (2015, January 5–7). Exploiting dependency relations for sentence level sentiment classification using SVM. Proceedings of the 2015 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT), Coimbatore, India.10.1109/ICECCT.2015.7226110 ↗
9. Raut, P.P., and Patil, N.N. (2015, January 2–4). Classification of controversial news article based on disputant relation by SVM classifier. Proceedings of the 2015 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions), Noida, India.10.1109/ICRITO.2015.7359277 ↗
10. Deep neural network-based relation extraction: An overview10.1007/s00521-021-06667-3 ↗
11. BERT gated multi-window attention network for relation extraction10.1016/j.neucom.2021.12.044 ↗
12. Wu, S., and He, Y. (2019, January 3–7). Enriching Pre-trained Language Model with Entity Information for Relation Classification. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.10.1145/3357384.3358119 ↗
13. Liang, D., Xu, W., and Zhao, Y. (2017, January 3). Combining Word-Level and Character-Level Representations for Relation Classification of Informal Text. Proceedings of the 2nd Workshop on Representation Learning for NLP, Vancouver, BC, Canada.10.18653/v1/W17-2606 ↗
14. Li, C., and Tian, Y. (2020). Downstream Model Design of Pre-trained Language Model for Relation Extraction Task. arXiv.
15. Ma, Y., Sun, Z., Zhang, D., and Feng, Y. (2022). Traditional Chinese Medicine Word Representation Model Augmented with Semantic and Grammatical Information. Information, 13.10.3390/info13060296 ↗
16. Nguyen, T.H., and Grishman, R. (2015, January 5). Relation Extraction: Perspective from Convolutional Neural Networks. Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, Denver, CO, USA.10.3115/v1/W15-1506 ↗
17. dos Santos, C.N., Xiang, B., and Zhou, B. (2015). Classifying Relations by Ranking with Convolutional Neural Networks. arXiv.10.3115/v1/P15-1061 ↗
18. Wang, L., Cao, Z., de Melo, G., and Liu, Z. (2016, January 7–12). Relation Classification via Multi-Level Attention CNNs. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany.10.18653/v1/P16-1123 ↗
19. Recurrent Neural Networks With External Addressable Long-Term and Working Memory for Learning Long-Term Dependences10.1109/TNNLS.2019.2910302 ↗
20. Li, J., Luong, M.-T., Jurafsky, D., and Hovy, E. (2015). When Are Tree Structures Necessary for Deep Learning of Representations?. arXiv.10.18653/v1/D15-1278 ↗
21. Zhang, D., and Wang, D. (2015). Relation Classification via Recurrent Neural Network. arXiv.
22. Lee, J., Seo, S., and Choi, Y.S. (2019). Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing. Symmetry, 11.10.3390/sym11060785 ↗
23. An Attentive Survey of Attention Models10.1145/3465055 ↗
24. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
25. Im, J., and Cho, S. (2017). Distance-based Self-Attention Network for Natural Language Inference. arXiv.
26. Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., and Zhang, C. (2017). DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding. arXiv.10.1609/aaai.v32i1.11941 ↗
27. Mnih, V., Heess, N., Graves, A., and Kavukcuoglu, K. (2014). Recurrent Models of Visual Attention. arXiv.
28. Jaderberg, M., Simonyan, K., and Zisserman, A. (2015, January 7–12). Spatial Transformer Networks. Proceedings of the Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, QC, Canada.
29. Hu, J., Shen, L., Albanie, S., Sun, G., and Wu, E. (2019). Squeeze-and-Excitation Networks. arXiv.10.1109/CVPR.2018.00745 ↗
30. Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., and Tang, X. (2017, January 21–26). Residual Attention Network for Image Classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017, Honolulu, HI, USA.10.1109/CVPR.2017.683 ↗
31. Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. (November, January 27). CCNet: Criss-Cross Attention for Semantic Segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision 2019, Seoul, Korea.10.1109/ICCV.2019.00069 ↗
32. Gao, Z., Xie, J., Wang, Q., and Li, P. (2019, January 15–20). Global Second-Order Pooling Convolutional Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00314 ↗
33. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., and Hu, Q. (2020). ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks. arXiv.10.1109/CVPR42600.2020.01155 ↗
34. Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., and Agrawal, A. (2018). Context Encoding for Semantic Segmentation. arXiv.10.1109/CVPR.2018.00747 ↗
35. Liu, Y., Shao, Z., Teng, Y., and Hoffmann, N. (2021). NAM: Normalization-based Attention Module. arXiv.
36. Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., and Xu, B. (2016, January 7–12). Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. Proceedings of the 54th Annual Meeting of the Association For Computational Linguistics, Berlin, Germany. Volume 2 Short Papers.10.18653/v1/P16-2034 ↗
37. Bidirectional LSTM with attention mechanism and convolutional layer for text classification10.1016/j.neucom.2019.01.078 ↗
38. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
39. Novel target attention convolutional neural network for relation classification10.1016/j.ins.2022.03.024 ↗
40. Enhance prototypical networks with hybrid attention and confusing loss function for few-shot relation classification10.1016/j.neucom.2022.04.067 ↗
41. Hybrid attention-based transformer block model for distant supervision relation extraction10.1016/j.neucom.2021.10.037 ↗
42. Lee, Y., Son, J., and Song, M. (2022). BertSRC: Transformer-based semantic relation classification. BMC Med. Inform. Decis. Mak., 22.10.1186/s12911-022-01977-5 ↗
43. Keya, A.J., Wadud, M.A., Mridha, M.F., Alatiyyah, M., and Hamid, M.A. (2022). AugFake-BERT: Handling Imbalance through Augmentation of Fake News Using BERT to Enhance the Performance of Fake News Classification. Appl. Sci., 12.10.3390/app12178398 ↗
44. Shao, C., Li, M., Li, G., Zhou, M., and Han, D. (2022). CRSAtt: By Capturing Relational Span and Using Attention for Relation Classification. Appl. Sci., 12.10.3390/app122111068 ↗
45. Woo, S., Park, J., Lee, J.-Y., and Kweon, I.S. (2018). CBAM: Convolutional Block Attention Module. arXiv.10.1007/978-3-030-01234-2_1 ↗
46. Hou, Q., Zhou, D., and Feng, J. (2021). Coordinate Attention for Efficient Mobile Network Design. arXiv.10.1109/CVPR46437.2021.01350 ↗
47. Liu, H.-I., Chen, M.-W., Kao, W.-C., Yeh, Y.-W., and Yang, C.-X. (2022, January 26–29). GSAP: A Hybrid GRU and Self-Attention Based Model for Dual Medical NLP Tasks. Proceedings of the 2022 14th International Conference on Knowledge and Smart Technology (KST), Chon buri, Thailand.10.1109/KST53302.2022.9727234 ↗
48. Jin, Y., Wu, D., and Guo, W. (2020). Attention-Based LSTM with Filter Mechanism for Entity Relation Classification. Symmetry, 12.10.3390/sym12101729 ↗
49. Laurent, C., Pereyra, G., Brakel, P., Zhang, Y., and Bengio, Y. (2015). Batch Normalized Recurrent Neural Networks. arXiv.10.1109/ICASSP.2016.7472159 ↗
50. Ioffe, S., and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv.
51. Hendrickx, I., Kim, S.N., Kozareva, Z., Nakov, P., Séaghdha, D.Ó., Padó, S., Pennacchiotti, M., Romano, L., and Szpakowicz, S. (2009, January 4). SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, DEW ’09, Boulder, CO, USA.10.3115/1621969.1621986 ↗