Paraphrase Identification with Lexical, Syntactic and Sentential Encodings
Source abstract
Abstract
Paraphrase identification has been one of the major topics in Natural Language Processing (NLP). However, how to interpret a diversity of contexts such as lexical and semantic information within a sentence as relevant features is still an open problem. This paper addresses the problem and presents an approach for leveraging contextual features with a neural-based learning model. Our Lexical, Syntactic, and Sentential Encodings (LSSE) learning model incorporates Relational Graph Convolutional Networks (R-GCNs) to make use of different features from local contexts, i.e., word encoding, position encoding, and full dependency structures. By utilizing the hidden states obtained by the R-GCNs as well as lexical and sentential encodings by Bidirectional Encoder Representations from Transformers (BERT), our model learns the contextual similarity between sentences effectively. The experimental results by using the two benchmark datasets, Microsoft Research Paraphrase Corpus (MRPC) and Quora Question Pairs (QQP) show that the improvement compared with the baseline, BERT sentential encodings model, was 1.7% F1-score on MRPC and 1.0% F1-score on QQP. Moreover, we verified that the combination of position encoding and syntactic features contributes to performance improvement.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Hangzhou Dianzi University, HangZhou 310018, China
Integrated Graduate School of Medicine, Engineering, and Agricultural Sciences, Faculty of Engineering, University of Yamanashi, Kofu 400-8511, Japan
Graduate Faculty of Interdisciplinary Research, University of Yamanashi, Kofu 400-8511, Japan
References (64)
Show source-provided references
1. He, H., Gimpel, K., and Lin, J. (2015, January 17–21). Multi-perspective Sentence Similarity Modeling with Convolutional Neural Networks. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1181 ↗
2. ABCNN: Attention-based Convolutional Neural Network for Modeling Sentence Pairs10.1162/tacl_a_00097 ↗
3. Liu, P., Qiu, X., and Huang, X. (2016). Modelling Interaction of Sentence Pair with Coupled-LSTMs. arXiv.10.18653/v1/D16-1176 ↗
4. Chen, Q., Zhu, X., Ling, Z., Wei, S., Jiang, H., and Inkpen, D. (August, January 30). Enhanced LSTM for Natural Language Inference. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada.
5. Wieting, J., and Gimpel, K. (August, January 30). Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada.
6. Wang, Y., Huang, H., Chong, F., Zhou, Q., Jiahui, G., and Xiong, G. (2016, January 7–12). CSE: Conceptual Sentence Embeddings based on Attention Model. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany.10.18653/v1/P16-1048 ↗
7. Oren, M., Jacob, G., and Ido, D. (2016, January 11–12). Context2vec: Learning Generic Context Embedding with Bidirectional LSTM. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany.
8. Sanjeev, A., Yingyu, L., and Tengyu, M. (2017, January 24–26). A Simple but Tough-to-Beat Baseline for Sentence Embeddings. Proceedings of the 5th International Conference on Learning Representations, Toulon, France.
9. Mark, N., Mohit, I., Matt, G., Christopher, C., Kenton, L., and Luke, Z. (2018, January 1–6). Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.
10. Cer, D., Yang, Y., Kong, S., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., and Tar, C. (2018). Univeral Sentence Encoder. arXiv.10.18653/v1/D18-2029 ↗
11. Jacob, D., Ming-Wei, C., Kenton, L., and Kristina, T. (2019, January 2–7). BERT: Pre-training on Deep Bidirectional Transfomers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
12. Michaël, D., Xavier, B., and Pierre, V. (2016, January 5–10). Convolutiona Neural Networks on Graphs with Fast Localized Sectral Filtering. Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain.
13. Thomas, K., and Max, W. (2017, January 24–26). SEMI-Supervised Classification with Graph Convolutional Networks. Proceedings of the 5th International Conference on Learning Representations, Toulon, France.
14. Felix, W., Tianyi, Z., de Souza, A.H., Christopher, F., Tao, Y., and Weinberger, K.Q. (2019, January 9–15). Simplifying Graph Convolutional Networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.
15. Socher, R., Bauer, J., Manning, C.D., and Ng, A.Y. (2018, January 15–20). Parsing with Compositional Vector Grammars. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.
16. Danqi, C., and Manning, C.D. (2014, January 25–29). A Fast and Accurate Dependency Parser using Neural Networks. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Doha, Qatar.
17. Socher, R., Huang, E.H., Pennington, J., Ng, A.Y., and Manning, C.D. (2011). Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. Advances in Neural Information Processing Systems, The MIT press.
18. Hu, B., Lu, Z., Li, H., and Chen, Q. (2015). Convolutional Neural Network Architectures for Matching Natural Language Sentences. Advances in Neural Information Processing Systems, The MIT press.
19. Tai, K.S., Socher, R., and Manning, C.D. (2015, January 26–31). Improved Semantic Representations from Tree-structured Long Short-term Memory Networks. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, Beijing, China.10.3115/v1/P15-1150 ↗
20. Language models are unsupervised multitask learners
21. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018). GLUE: A Multi-task Benchmark and Analysis Platform for Natural Language Understanding. arXiv.10.18653/v1/W18-5446 ↗
22. Lample, G., and Conneau, A. (2019). Cross-lingual Language Model Pretraining. arXiv.
23. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, The MIT Press.
24. Subramanian, S., Trischler, A., Bengio, Y., and Pal, J.C. (May, January 30). Learning General Purpose Distributed Sentence representations via Large Scale Multi-task Learning. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada.
25. Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., and Wang, H. (2019). ERNIE 2.0: A Continual Pre-training Framework for Language Understanding. arXiv.10.1609/aaai.v34i05.6428 ↗
26. Liu, X., He, P., Chen, W., and Gao, J. (2019). Multi-task Deep Neural Networks for Natural Language Understanding. arXiv.10.18653/v1/P19-1441 ↗
27. Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Yu, P.S. (2019). A Comprehensive Survey on Graph Neural Networks. arXiv.
28. Michael, S., Nipf, T.N., Peter, B., Rianne, V.D.B., Ivan, T., and Max, W. (2018, January 3–7). Modeling Relational Data with Graph Convolutional Networks. Proceedings of the European Semantic Web Conference, Crete, Greece.
29. Densely Connected Graph Convolutional Networks for Graph-to-Sequence Learning10.1162/tacl_a_00269 ↗
30. Joost, B., Ivan, T., Wilker, A., Diego, M., and Khalil, S. (2017, January 7–11). Graph Convolutional Encoders for Syntax-aware Neural Machine Translation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.
31. Beck, D., Haffari, G., and Cohn, T. (2018, January 15–20). Graph-to-Sequence Learning using Gated Graph Neural Networks. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1026 ↗
32. Yinchuan, X., and Junlin, Y. (2019, January 2). Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution. Proceedings of the 1st Workshop on Gender Bias in Natural Language Processing, Florence, Italy.
33. Zhijiang, G., Yan, Z., and Wei, L. (August, January 28). Attention Guided Graph Convolutional Networks for Relation Extraction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
34. Diego, M., and Ivan, T. (2017, January 7–11). Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.
35. Hamilton, W.L., Ying, R., and Leskovec, J. (2017, January 4–9). Inductive representation learning on large graphs. Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
36. Peter, V., Guillem, C., Arantxa, C., Adriana, R., Pietro, L., and Yoshua, B. (May, January 30). Graph Attention Networks. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
37. Liang, Y., Chengsheng, M., and Yuan, L. (February, January 27). Graph Convolutional Networks for Text Classification. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
38. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., and Norouzi, M. (2016). Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv.
39. Shikhar, V., Manik, B., Prateek, Y., Piyush, R., Chiranjib, B., and Partha, T. (August, January 28). Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
40. Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., and Graves, A. (2016, January 5–10). Conditional Image Generation with PixelCNN Decoders. Proceedings of the 30th Conference on Neural Information Processin System, Barcelona, Spain.
41. Li, Q., Han, Z., and Wu, X.M. (2018, January 2–7). Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning. Proceedings of the 32nd AAAI conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11604 ↗
42. Yang, L., Kang, Z., Can, X., Jin, D., Yang, B., and Guo, Y. (2019, January 10–16). Topology Optimization based Graph Convolutional Network. Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2019/563 ↗
43. Kingma, D.P., and Ba, J. (2015, January 7–9). ADAM: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA.
44. Dolan, W.B., and Brockett, C. (2005, January 14). Automatically Constructing a Corpus of Sentential Paraphrases. Proceedings of the Third International Workshop on Paraphrasing, Jeju Island, Korea.
45. Shankar, I., Nikhil, D., and Kornél, C. (2020, March 01). First Quora Dataset Release: Question Pairs. Available online: https://data.quora.com/First-Quora-Dataset-Releasee-Question-Pairs.
46. Zhiguo, W., Wael, H., and Radu, F. (2017, January 19–25). Bilateral Multi-Perspective Matching for Natural Language Sentences. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne, Australia.
47. Phang, J., Fevry, T., and Bowman, S.R. (2019). Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks. arXiv.
48. Liang, C., Paritosh, P.K., Rajendran, V., and Forbus, K.D. (2016, January 9–15). Learning Paraphrase Identification with Structural Alignment. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, NY, USA.
49. Subramanian, S., Trischler, A., Bengio, Y., and Pal, C.J. (2018). Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning. arXiv.
50. Yuki, A., and Junichi, T. (2019, January 3–7). Transfer Fine-Tuning: A BERT Case Study. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China.
51. Wei, W., Bin, B., Ming, Y., Chen, W., Zuyi, B., Liwei, P., and Luo, S. (2019). StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding. arXiv.
52. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
53. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (May, January 26). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia.
54. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P.J. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv.
55. Chen, Z., Yu, C., Zhe, G., Siqi, S., Thomas, G., and Jing, L. (2019). FreeLB: Enhanced Adversarial Training for Natural Language Understanding. arXiv.
56. Clark, K., Luong, M.T., Le, Q.V., and Manning, C.D. (May, January 26). ELECTRA: Pre-Training Text Encoders as Discriminators Rather than Generators. Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia.
57. Nie, Y., and Bansal, M. (, January September). Shortcut-stacked Sentence Encoders for Multi-Domain Inference. Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP, Copenhagen, Denmark.
58. Wuwei, L., and Wei, X. (2018, January 20–26). Neural Network Models for Paraphrase Identification Semantic Textual Similarity Natural Language Inference and Question Answering. Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, NM, USA.
59. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017, January 7–11). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1070 ↗
60. He, H., and Lin, J. (2016, January 12–17). Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1108 ↗
61. Parikh, A., Tom, O., Das, D., and Uszkoreit, J. (2016, January 1–5). A Decomposable Attention Model for Natural Language Inference. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1244 ↗
62. Moschitti, A. (2006, January 18–22). Efficient Convolution Kernels for Dependency and Constituent Syntactic Trees. Proceedings of the 17th European Conference on Machine Learning and the 10th European Conference on Principles and Practice of Knowledge Discovery in Databases, Berlin, Germany.
63. Moschitti, A., Chu-Carroll, J., Patwardhan, S., Fan, J., and Riccardi, G. (2011, January 27–31). Using Syntactic and Semantic Structural Kernels for Classifying Definition Questions in Jeopardy!. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Scotland, UK.
64. Papernot, N., Abadi, M., Úlfar, E., Goodfellow, I., and Talwar, K. (2017, January 24–26). Semi-Supervised Knowledge Transfer for Deep Learning from Private Training Data. Proceedings of the International Conference on Learning Representations, Toulon, France.