A Sentence-Matching Model Based on Multi-Granularity Contextual Key Semantic Interaction
Source abstract
Abstract
In the task of matching Chinese sentences, the key semantics within sentences and the deep interaction between them significantly affect the matching performance. However, previous studies mainly relied on shallow interactions based on a single semantic granularity, which left them vulnerable to interference from overlapping terms. It is particularly challenging to distinguish between positive and negative examples within datasets from the same thematic domain. This paper proposes a sentence-matching model that incorporates multi-granularity contextual key semantic interaction. The model combines multi-scale convolution and multi-level convolution to extract different levels of contextual semantic information at word, phrase, and sentence granularities. It employs multi-head self-attention and cross-attention mechanisms to align the key semantics between sentences. Furthermore, the model integrates the original, similarity, and dissimilarity information of sentences to establish deep semantic interaction. Experimental results on both open- and closed-domain datasets demonstrate that the proposed model outperforms existing baseline models in terms of matching performance. Additionally, the model achieves matching effectiveness comparable to large-scale pre-trained language models while utilizing a lightweight encoder.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Text Readability and Simplification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China
Computer Technology Application Key Lab of the Yunnan Province, Kunming 650500, China
Funding
Key projects of science and technology plan of Yunnan Provincial Department of Science and Technology
References (36)
Show source-provided references
1. Garg, S., and Ramakrishnan, G. (2020, January 16–20). BAE: BERT-based Adversarial Examples for Text Classification. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.498 ↗
2. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
3. Reimers, N., and Gurevych, I. (2019, January 3–7). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1410 ↗
4. Wang, S., and Jiang, J. (2017, January 24–26). A compare-aggregate model for matching text sequences. Proceedings of the ICLR 2017: International Conference on Learning Representations, Toulon, France.
5. Short text matching model with multiway semantic interaction based on multi-granularity semantic embedding10.1007/s10489-022-03410-w ↗
6. Wang, M., He, X., Liu, Y., Qing, L., Zhang, Z., and Chen, H. (2023). MAGE: Multi-scale context-aware interaction based on multi-granularity embedding for chinese medical question answer matching. Comput. Methods Programs Biomed., 228.10.1016/j.cmpb.2022.107249 ↗
7. Sun, Z., Li, X., Sun, X., Meng, Y., Ao, X., He, Q., Wu, F., and Li, J. (2021, January 1–6). ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Virtual.10.18653/v1/2021.acl-long.161 ↗
8. Zhang, S., Zhang, X., Wang, H., Cheng, J., Li, P., and Ding, Z. (2017). Chinese medical question answer matching using end-to-end character-level multi-scale CNNs. Appl. Sci., 7.10.3390/app7080767 ↗
9. BERT-SMAP: Paying attention to Essential Terms in passage ranking beyond BERT10.1016/j.ipm.2021.102788 ↗
10. MKPM: Multi keyword-pair matching for natural language sentences10.1007/s10489-021-02306-5 ↗
11. Zhao, S., Huang, Y., Su, C., Li, Y., and Wang, F. (2020, January 17–20). Interactive attention networks for semantic text matching. Proceedings of the 2020 IEEE International Conference on Data Mining (ICDM), Sorrento, Italy.10.1109/ICDM50108.2020.00095 ↗
12. Fei, H., Ren, Y., and Ji, D. (2020, January 16–20). Improving text understanding via deep syntax-semantics communication. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Online.10.18653/v1/2020.findings-emnlp.8 ↗
13. Zhang, X., Li, Y., Lu, W., Jian, P., and Zhang, G. (2020, January 8–13). Intra-correlation encoding for Chinese sentence intention matching. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online).10.18653/v1/2020.coling-main.455 ↗
14. A simple and efficient text matching model based on deep interaction10.1016/j.ipm.2021.102738 ↗
15. Wang, X., and Yang, H. (2022). MGMSN: Multi-Granularity Matching Model Based on Siamese Neural Network. Front. Bioeng. Biotechnol., 10.10.3389/fbioe.2022.839586 ↗
16. MatchACNN: A multi-granularity deep matching model10.1007/s11063-022-11047-6 ↗
17. Deep feature fusion model for sentence semantic matching
18. Lai, Y., Feng, Y., Yu, X., Wang, Z., Xu, K., and Zhao, D. (February, January 27). Lattice cnns for matching based chinese question answering. Proceedings of the Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
19. Chen, L., Zhao, Y., Lyu, B., Jin, L., Chen, Z., Zhu, S., and Yu, K. (2020, January 5–10). Neural graph matching networks for Chinese short text matching. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.547 ↗
20. Zhang, X., Lu, W., Zhang, G., Li, F., and Wang, S. (2020, January 11–14). Chinese sentence semantic matching based on multi-granularity fusion model. Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Singapore.10.1007/978-3-030-47436-2_19 ↗
21. Zhao, P., Lu, W., Li, Y., Yu, J., Jian, P., and Zhang, X. (2021, January 18–22). Chinese semantic matching with multi-granularity alignment and feature fusion. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9534130 ↗
22. Lyu, B., Chen, L., Zhu, S., and Yu, K. (2021, January 2–9). Let: Linguistic knowledge enhanced graph transformer for chinese short text matching. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i15.17592 ↗
23. Tao, H., Tong, S., Zhang, K., Xu, T., Liu, Q., Chen, E., and Hou, M. (2021, January 2–9). Ideography leads us to the field of cognition: A radical-guided associative model for chinese text classification. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i15.17637 ↗
24. Multi-granularity interaction model based on pinyins and radicals for Chinese semantic matching10.1007/s11280-022-01037-y ↗
25. Wu, Z., Liang, J., Zhang, Z., and Lei, J. (2021). Exploration of text matching methods in Chinese disease Q&A systems: A method using ensemble based on BERT and boosted tree models. J. Biomed. Inform., 115.10.1016/j.jbi.2021.103683 ↗
26. BMCSA: Multi-feature spatial convolution semantic matching model based on BERT10.3233/JIFS-212624 ↗
27. Zou, Y., Liu, H., Gui, T., Wang, J., Zhang, Q., Tang, M., Li, H., and Wang, D. (2022, January 22–27). Divide and Conquer: Text Semantic Matching with Disentangled Keywords and Intents. In Proceeding of the Findings of Computational Linguistics: ACL 2022, Dublin, Ireland.10.18653/v1/2022.findings-acl.287 ↗
28. Liu, X., Chen, Q., Deng, C., Zeng, H., Chen, J., Li, D., and Tang, B. (2018, January 20–26). Lcqmc: A large-scale chinese question matching corpus. Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, NM, USA.
29. Chen, J., Chen, Q., Liu, X., Yang, H., Lu, D., and Tang, B. (November, January 31). The bq corpus: A large-scale domain-specific chinese corpus for sentence semantic equivalence identification. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1536 ↗
30. Text-attentional convolutional neural network for scene text detection10.1109/TIP.2016.2547588 ↗
31. Siami-Namini, S., Tavakoli, N., and Namin, A.S. (2019, January 9–12). The performance of LSTM and BiLSTM in forecasting time series. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.10.1109/BigData47090.2019.9005997 ↗
32. Wang, Z., Hamza, W., and Florian, R. (2017, January 19–25). Bilateral multi-perspective matching for natural language sentences. Proceedings of the International Joint Conference on Artificial Intelligence, Melbourne, Australia.10.24963/ijcai.2017/579 ↗
33. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Deeply-recursive convolutional network for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.181 ↗
34. Pre-training with whole word masking for chinese bert10.1109/TASLP.2021.3124365 ↗
35. Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., and Lu, Y. (2021). Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv.
36. Bhojanapalli, S., Yun, C., Rawat, A.S., Reddi, S., and Kumar, S. (2020, January 13–18). Low-rank bottleneck in multi-head attention models. Proceedings of the International Conference on Machine Learning, Virtual.