Short-Text Semantic Similarity (STSS): Techniques, Challenges and Future Perspectives
Source abstract
Abstract
In natural language processing, short-text semantic similarity (STSS) is a very prominent field. It has a significant impact on a broad range of applications, such as question–answering systems, information retrieval, entity recognition, text analytics, sentiment classification, and so on. Despite their widespread use, many traditional machine learning techniques are incapable of identifying the semantics of short text. Traditional methods are based on ontologies, knowledge graphs, and corpus-based methods. The performance of these methods is influenced by the manually defined rules. Applying such measures is still difficult, since it poses various semantic challenges. In the existing literature, the most recent advances in short-text semantic similarity (STSS) research are not included. This study presents the systematic literature review (SLR) with the aim to (i) explain short sentence barriers in semantic similarity, (ii) identify the most appropriate standard deep learning techniques for the semantics of a short text, (iii) classify the language models that produce high-level contextual semantic information, (iv) determine appropriate datasets that are only intended for short text, and (v) highlight research challenges and proposed future improvements. To the best of our knowledge, we have provided an in-depth, comprehensive, and systematic review of short text semantic similarity trends, which will assist the researchers to reuse and enhance the semantic information.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer and Information Sciences, Universiti Teknologi PETRONAS, Seri Iskandar 32160, Malaysia
Department of Information Technology, University of Sindh, Jamshoro 71000, Pakistan
Department of Artificial Intelligence, Tomas Bata University, 760 01 Zlín, Czech Republic
Funding
Yayasan UTP Pre-commercialization grant (YUTP-PRG)
References (100+)
Show source-provided references
1. A short text classification method based on convolutional neural network and semantic extension10.2991/ijcis.d.201207.001 ↗
2. Zhao, H., Hu, G., and Jiao, C. (2017, January 10–11). Short Text Similarity Calculation Using Semantic Information. Proceedings of the 2017 3rd International Conference on Big Data Computing and Communications (BIGCOM), Chengdu, China.
3. Management, Paraphrase identification and semantic text similarity analysis in Arabic news tweets using lexical, syntactic, and semantic features10.1016/j.ipm.2017.01.002 ↗
4. Olowolayemo, A., Nawi, S.D., and Mantoro, T. (2018, January 6–8). Short, answer scoring in English grammar using text similarity measurement. Proceedings of the 2018 International Conference on Computing, Engineering and Design (ICCED), Bangkok, Thailand.10.1109/ICCED.2018.00034 ↗
5. Keywords extraction with the deep neural network model10.1016/j.neucom.2019.11.083 ↗
6. Hua, W., Wang, Z., Wang, H., Zheng, K., and Zhou, X. (2015, January 13–17). Short text understanding through lexical-semantic analysis. Proceedings of the 2015 IEEE 31st International Conference on Data Engineering, Seoul, Republic of Korea.10.1109/ICDE.2015.7113309 ↗
7. A survey on the techniques, applications and performance of short text semantic similarity10.1002/cpe.5971 ↗
8. A scoring rubric for automatic short answer grading system10.12928/telkomnika.v17i2.11785 ↗
9. Neuroscience, Short-Text Classification Detector: A Bert-Based Mental Approach
10. A study of using syntactic cues in the short-text similarity measure
11. Challenges of Short Sentence Writing Encountered by First-Year Saudi EFL Undergraduate Students10.24093/awej/vol13no1.35 ↗
12. Gaddipati, S.K. (2021). Automatic Formative Assessment for Students’ Short Text Answers through Feature Extraction. [Ph.D. Thesis, Hochschule Bonn-Rhein-Sieg].
13. State-of-the-art IoV trust management a meta-synthesis systematic literature review (SLR)10.7717/peerj-cs.334 ↗
14. A systematic review for smart city data analytics10.1145/3239566 ↗
15. Kitchenham, B.J.K. (2004). Procedures for Performing Systematic Reviews, Keele University.
16. Shih, S.-H., and Yeh, C. (2019, January 23–25). A Short Answer Grading System in Chinese by CNN. Proceedings of the 10th International Conference on Awareness Science and Technology (iCAST), Morioka, Japan.
17. Incorporating context-relevant concepts into convolutional neural networks for short text classification10.1016/j.neucom.2019.08.080 ↗
18. Perera, N., Priyankara, C., and Jayasekara, D. (2020, January 4–7). Identifying Irrelevant Answers in Web Based Question Answering Systems. Proceedings of the 20th International Conference on Advances in ICT for Emerging Regions (ICTer), Colombo, Sri Lanka.10.1109/ICTer51097.2020.9325449 ↗
19. Deep learning for short answer scoring
20. Wang, P., Xu, J., and Xu, B. (2017). Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Association for Computational Linguistics.
21. Liu, J., Ma, H., Xie, X., and Cheng, J.J.E. (2022). Short Text Classification for Faults Information of Secondary Equipment Based on Convolutional Neural Networks. Energies, 15.10.3390/en15072400 ↗
22. Hu, Y., Li, Y., Yang, T., and Pan, Q. (2018, January 18–21). Short text classification with a convolutional neural networks based method. Proceedings of the 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore.10.1109/ICARCV.2018.8581332 ↗
23. Management, A deep network model for paraphrase detection in short text messages10.1016/j.ipm.2018.06.005 ↗
24. Unlabeled short text similarity with LSTM encoder10.1109/ACCESS.2018.2885698 ↗
25. Dwivedi, V.P., Singh, D.K., and Jha, S. (2017, January 14–16). Gender classification of blog authors: With feature engineering and deep learning using LSTM networks. Proceedings of the 9th International Conference on Advanced Computing (ICoAC), Chennai, India.10.1109/ICoAC.2017.8441506 ↗
26. Li, Q., Wu, Q., Zhu, C., and Zhang, J. (2019, January 20–25). Bi-level masked multi-scale CNN-RNN networks for short text representation. Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), Sydney, NSW, Australia.10.1109/ICDAR.2019.00147 ↗
27. Edo-Osagie, O., and Lake, I.L. (2019, January 12–14). Attention-based recurrent neural networks (RNNs) for short text classification: An application in public health monitoring. Proceedings of the 15th International Work-Conference on Artificial Neural Networks, IWANN 2019, Gran Canaria, Spain.10.1007/978-3-030-20521-8_73 ↗
28. State-of-the-Art: Assessing Semantic Similarity in Automated Short-Answer Grading Systems
29. Lee, J.Y., and Dernoncourt, F. (2016). Sequential short-text classification with recurrent and convolutional neural networks. arXiv.10.18653/v1/N16-1062 ↗
30. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
31. Mozafari, J., and Fatemi, A. (2019). BAS: An answer selection method using BERT language model. arXiv.
32. Automatic Short Answer Grading System in Indonesian Language Using BERT Machine Learning
33. Luo, J. (2021). Automatic Short Answer Grading Using Deep Learning. [Ph.D. Thesis, Illinois State University].
34. Alammary, A.S. (2022). BERT Models for Arabic Text Classification: A Systematic Review. Appl. Sci., 12.10.3390/app12115720 ↗
35. Short answer grading using contextual word embedding and linear regression
36. Gaddipati, S.K. (2020). Comparative evaluation of pretrained transfer learning models on automatic short answer grading. arXiv.
37. Garg, J., Papreja, J., Apurva, K., and Jain, G. (2022, January 24–26). Domain-Specific Hybrid BERT based System for Automatic Short Answer Grading. Proceedings of the 2nd International Conference on Intelligent Technologies (CONIT), Hubli, India.10.1109/CONIT55038.2022.9847754 ↗
38. Automatic Short-Answer Grading via BERT-Based Deep Neural Networks10.1109/TLT.2022.3175537 ↗
39. The eras and trends of automatic short answer grading10.1007/s40593-014-0026-8 ↗
40. Mohler, M. (2011, January 19–24). Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA.
41. Wang, Z., Lan, A.S., and Waters, A. (2019). A Meta-Learning Augmented Bidirectional Transformer Model for Automatic Short Answer Grading, EDM.
42. Sung, C. (2019). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics.
43. Bi-GRU Urgent Classification for MOOC Discussion Forums Based on BERT10.1109/ACCESS.2021.3072734 ↗
44. Camus, L., and Filighera, A. (2020, January 6–10). Investigating transformers for automatic short answer grading. Proceedings of the International Conference on Artificial Intelligence in Education, Ifrane, Morocco.10.1007/978-3-030-52240-7_8 ↗
45. Sung, C., Dhamecha, T.I., and Mukhi, N. (2019, January 25–29). Improving short answer grading using transformer-based pre-training. Proceedings of the International Conference on Artificial Intelligence in Education, Chicago, IL, USA.10.1007/978-3-030-23204-7_39 ↗
46. Mayfield, E., and Black, A.W. (2020, January 9). Should you fine-tune BERT for automated essay scoring?. Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications, Seattle, WA, USA.10.18653/v1/2020.bea-1.15 ↗
47. Nie, F., Zhou, S., Liu, J., and Wang, J. (November, January 31). Aggregated semantic matching for short text entity linking. Proceedings of the 22nd Conference on Computational Natural Language Learning, Brussels, Belgium.
48. De Boom, C. (2015, January 14–17). Learning semantic similarity for very short texts. Proceedings of the International Conference on Data Mining Workshop (ICDMW), Atlantic City, NJ, USA.10.1109/ICDMW.2015.86 ↗
49. Short text similarity measurement methods: A review10.1007/s00500-020-05479-2 ↗
50. Measuring the short text similarity based on semantic and syntactic information10.1016/j.future.2020.07.043 ↗
51. Ye, Z., and Jiang, G. (2020). ECAI 2020, IOS Press.
52. Hu, W., Dang, A., and Tan, Y. (2019, January 26–30). A survey of state-of-the-art short text matching algorithms. Proceedings of the International Conference on Data Mining and Big Data, Chiang Mai, Thailand.10.1007/978-981-32-9563-6_22 ↗
53. Short-text intention recognition based on multi-dimensional dynamic word vectors
54. Engineering, A self-attention based neural architecture for Chinese medical named entity recognition10.3934/mbe.2020197 ↗
55. Lin, X., Xiong, G., Gou, G., and Li, Z. (2022). Proceedings of the ACM Web Conference 2022, Association for Computing Machinery.
56. Beltagy, I., Lo, K., and Cohan, A.J. (2019). SciBERT: A pre-trained language model for scientific text. arXiv.10.18653/v1/D19-1371 ↗
57. Devlin, J., Chang, M.-W., and Lee, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
58. Enriching word vectors with subword information10.1162/tacl_a_00051 ↗
59. Sedoc, J., and Ungar, L. (2019). Proceedings of the First Workshop on Gender Bias in Natural Language Processing, Association for Computational Linguistics.
60. Detecting formal thought disorder by deep contextualized word representations10.1016/j.psychres.2021.114135 ↗
61. Fernandez, N., Ghosh, A., Liu, N., Wang, Z., Choffin, B., Baraniuk, R., and Lan, A.J. (2022, January 27–31). Automated Scoring for Reading Comprehension via In-context BERT Tuning. Proceedings of the Artificial Intelligence in Education: 23rd International Conference, AIED 2022, Durham, UK.10.1007/978-3-031-11644-5_69 ↗
62. Li, Y., Yang, Y., Hu, Q., and Chen, C. (April, January 28). An Argument Extraction Decoder in Open Information Extraction. Proceedings of the Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event.10.1007/978-3-030-72113-8_21 ↗
63. Deep entity linking via eliminating semantic ambiguity with BERT
64. Kovaleva, O., and Romanov, A. (2019). Revealing the dark secrets of BERT. arXiv.10.18653/v1/D19-1445 ↗
65. Impact analysis of keyword extraction using contextual word embedding10.7717/peerj-cs.967 ↗
66. Tang, M., Gandhi, P., and Kabir, M. (2019). Progress notes classification and keyword extraction using attention-based deep learning models with BERT. arXiv.
67. Lyu, B., and Chen, L. (2021, January 2–9). Let: Linguistic knowledge enhanced graph transformer for chinese short text matching. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event.10.1609/aaai.v35i15.17592 ↗
68. Context-based feature technique for sarcasm identification in benchmark datasets using deep learning and BERT model10.1109/ACCESS.2021.3068323 ↗
69. Wiedemann, G., and Remus, S. (2019). Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings. arXiv.
70. Zhou, Y., Li, C., He, S., Wang, X., and Qiu, Y. (2019, January 1–3). Pre-trained contextualized representation for Chinese conversation topic classification. Proceedings of the 2019 IEEE International Conference on Intelligence and Security Informatics (ISI), Shenzhen, China.10.1109/ISI.2019.8823172 ↗
71. Vaswani, A., Shazeer, N., and Parmar, N. (2017, January 4–9). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
72. Heidari, M., Jones, J.H., and Uzuner, O. (2020, January 17–20). Deep contextualized word embedding for text-based online user profiling to detect social bots on Twitter. Proceedings of the 2020 International Conference on Data Mining Workshops (ICDMW), Sorrento, Italy.10.1109/ICDMW51313.2020.00071 ↗
73. Amur, Z.H., Hooi, Y.K., and Soomro, G.M. (2022). 2022 International Conference on Digital Transformation and Intelligence (ICDI), IEEE.
74. Mu, J. (2017). All-but-the-top: Simple and effective postprocessing for word representations. arXiv.
75. Zhang, Z., and Wu, Y. (2020, January 7–12). Semantics-aware BERT for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
76. Word embeddings for biomedical natural language processing: A survey10.1111/lnc3.12402 ↗
77. Peters, M.E., Neumann, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
78. Gupta, H., and Patel, M. (2020, January 7–9). Study of extractive text summarizer using the Elmo embedding. Proceedings of the Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Palladam, India.10.1109/I-SMAC49090.2020.9243610 ↗
79. Liu, C., Gao, Y., Sun, L., Feng, J., Yang, H., and Ao, X. (2022). Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery.
80. A machine learning approach to predicting psychosis using semantic density and latent content analysis10.1038/s41537-019-0077-9 ↗
81. Naseem, U., Musial, K., Eklund, P., and Prasad, M. (2020, January 19–24). Biomedical named-entity recognition by hierarchically fusing biobert representations and deep contextual-level word-embedding. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9206808 ↗
82. Amur, Z.H., and Hooi, Y. (2022, January 17–18). State-of-the Art: Short Text Semantic Similarity (STSS) Techniques in Question Answering Systems (QAS). Proceedings of the International Conference on Artificial Intelligence for Smart Community, Seri Iskandar, Malaysia.10.1007/978-981-16-2183-3_98 ↗
83. Galhardi, L.B., and Brancher, J.D. (2018, January 13–16). Machine learning approach for automatic short answer grading: A systematic review. Proceedings of the Advances in Artificial Intelligence-IBERAMIA 2018: 16th Ibero-American Conference on AI, Trujillo, Peru.10.1007/978-3-030-03928-8_31 ↗
84. Zhang, Y., and Shah, R. (2016). Deep Learning + Student Modeling + Clustering: A Recipe for Effective Automatic Short Answer Grading, Institute of Education Sciences.
85. Saha, S., Dhamecha, T.I., Marvaniya, S., Sindhgatta, R., and Sengupta, B. (2018, January 27–30). Sentence level or token level features for automatic short answer grading? Use both. Proceedings of the Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK.10.1007/978-3-319-93843-1_37 ↗
86. Li, Z., Tomar, Y., and Passonneau, R.J. (2021, January 7–11). A Semantic Feature-Wise Transformation Relation Network for Automatic Short Answer Grading. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.487 ↗
87. Applications, Automatic short answer scoring based on paragraph embeddings
88. Al-Bataineh, H., and Farhan, W. (2019, January 4–6). Deep contextualized pairwise semantic similarity for Arabic language questions. Proceedings of the 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), Portland, OR, USA.10.1109/ICTAI.2019.00229 ↗
89. Yang, Y., Yuan, S., and Cer, D. (2018). Learning semantic textual similarity from conversations. arXiv.10.18653/v1/W18-3022 ↗
90. A set of Arabic word embedding models for use in Arabic NLP10.1016/j.procs.2017.10.117 ↗
91. Neelakantan, A., and Shankar, J. (2015). Efficient non-parametric estimation of multiple embeddings per word in vector space. arXiv.10.3115/v1/D14-1113 ↗
92. Word2Vec10.1017/S1351324916000334 ↗
93. Wieting, J., Bansal, M., and Gimpel, K. (2016). Charagram: Embedding words and sentences via character n-grams. arXiv.10.18653/v1/D16-1157 ↗
94. Laskar, M.T.R., Huang, X., and Hoque, E. (2020, January 11–16). Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France.
95. Reimers, N., and Gurevych, I. (2019). Alternative weighting schemes for elmo embeddings. arXiv.
96. Liu, L., Ren, X., Shang, J., and Peng, J. (2018). Efficient contextualized representation: Language model pruning for sequence labeling. arXiv.10.18653/v1/D18-1153 ↗
97. Walker Orr, J., and Tadepalli, P. (2018). Event Detection with Neural Networks: A Rigorous Empirical Evaluation. arXiv.10.18653/v1/D18-1122 ↗
98. Reimers, N., and Gurevych, I. (2018). Why comparing single performance scores does not allow to draw conclusions about machine learning approaches. arXiv.
99. Language models are unsupervised multitask learners
100. Vig, J., and Belinkov, Y. (2019). Analyzing the structure of attention in a transformer language model. arXiv.10.18653/v1/W19-4808 ↗