An Overview of Recent Advances in Natural Language Processing for Information Systems
Source abstract
Abstract
The crux of information systems is efficient storage and access to useful data by users. This paper is an overview of work that has advanced the use of such systems in recent years, primarily in machine learning, and specifically, deep learning methods. Situating progress in terms of classical pattern recognition techniques for text, we review computational methods to process spoken and written data. Digital assistants such as Siri, Cortana, and Google Now exploit large language models and encoder-only transformer-based systems such as BERT. Practical tasks include Machine Translation, Information Retrieval, Text Summarization, Question-Answering, Sentiment Analysis, Natural Language Generation, Named Entity Recognition, and Relation Extraction. Issues to be covered include: post-training through alignment, parsing, and Reinforcement Learning.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Sentiment Analysis and Opinion Mining · Big Data and Digital Economy
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Centre Énergie, Matériaux et Télécommunications (EMT), Institut National de la Recherche Scientifique (INRS), University of Quebec, Montreal, QC H5A 1K6, Canada
References (100+)
Show source-provided references
1. A survey on conversational recommender systems10.1145/3453154 ↗
2. Multimodal intelligence: Representation learning, information fusion, and applications10.1109/JSTSP.2020.2987728 ↗
3. Liu, Q., Kusner, M.J., and Blunsom, P. (2020). A survey on contextual embeddings. arXiv.
4. Zhao, H., Phung, D., Huynh, V., Jin, Y., Du, L., and Buntine, W. (2021). Topic modelling meets deep neural networks: A survey. arXiv.10.24963/ijcai.2021/638 ↗
5. Long, L., Wang, R., Xiao, R., Zhao, J., Ding, X., Chen, G., and Wang, H. (2024). On LLMs-driven synthetic data generation, curation, and evaluation: A survey. arXiv.10.18653/v1/2024.findings-acl.658 ↗
6. Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., and Liu, H. (2024). Large language models for data annotation and synthesis: A survey. arXiv.10.18653/v1/2024.emnlp-main.54 ↗
7. A survey on neural topic models: Methods, applications, and challenges10.1007/s10462-023-10661-7 ↗
8. Question answering at the cross-language evaluation forum 2003–2010
9. Progress in machine translation10.1016/j.eng.2021.03.023 ↗
10. Automatic text summarization: A comprehensive survey10.1016/j.eswa.2020.113679 ↗
11. A survey on sentiment analysis methods, applications, and challenges10.1007/s10462-022-10144-1 ↗
12. Mining fall-related information in clinical notes: Comparison of rule-based and novel word embedding-based machine learning approaches10.1016/j.jbi.2019.103103 ↗
13. Angeli, G., Liang, P., and Klein, D. (2010, January 9–11). A simple domain-independent probabilistic approach to generation. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Cambridge, MA, USA.
14. Natural language generation for electronic health records10.1038/s41746-018-0070-0 ↗
15. Zhang, H., Cai, J., Xu, J., and Wang, J. (2019, January 3–4). Pretraining-based natural language generation for text summarization. Proceedings of the Conference on Computational Natural Language Learning, Hong Kong, China.10.18653/v1/K19-1074 ↗
16. Mille, S., Belz, A., Bohnet, B., Graham, Y., Pitler, E., and Wanner, L. (2018, January 19). The first multilingual surface realisation shared task (SR’18): Overview and evaluation results. Proceedings of the Workshop on Multilingual Surface Realisation, Melbourne, Australia.10.18653/v1/W18-3601 ↗
17. Adapted large language models can outperform medical experts in clinical text summarization10.1038/s41591-024-02855-5 ↗
18. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Sauvestre, R., and Remez, T. (2023). Code LLaMa: Open foundation models for code. arXiv.
19. Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., and Li, Y.K. (2024). DeepSeek-Coder: When the Large Language Model Meets Programming--The Rise of Code Intelligence. arXiv.
20. A survey of joint intent detection and slot filling models in natural language understanding10.1145/3547138 ↗
21. Berant, J., Chou, A., Frostig, R., and Liang, P. (2013, January 18–21). Semantic parsing on freebase from question-answer pairs. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.10.18653/v1/D13-1160 ↗
22. Introduction to the special issue on word sense disambiguation: The state of the art
23. Pan, J.Z. (2009). Resource description framework. Handbook on Ontologies, Springer.10.1007/978-3-540-92673-3_3 ↗
24. Complex knowledge base question answering: A survey10.1109/TKDE.2022.3223858 ↗
25. Exploring AI-driven approaches for unstructured document analysis and future horizons10.1186/s40537-024-00948-z ↗
26. Mielke, S.J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W.Y., and Sagot, B. (2021). Between words and characters: A brief history of open-vocabulary modeling and tokenization in MLP. arXiv.
27. On structuring probabilistic dependences in stochastic language modelling10.1006/csla.1994.1001 ↗
28. An empirical study of smoothing techniques for language modeling10.1006/csla.1999.0128 ↗
29. The population frequencies of species and the estimation of population parameters10.1093/biomet/40.3-4.237 ↗
30. Gelsema, E.S., and Kanal, L.N. (1980). Interpolated estimation of Markov source parameters from sparse data. Pattern Recognition ill Practice, North-Holland.
31. Estimation of probabilities from sparse data for the language model component of a speech recognizer10.1109/TASSP.1987.1165125 ↗
32. Kneser, R., and Ney, H. (1995, January 9–12). Improved backing-off for m-gram language modeling. Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Detroit, MI, USA.
33. We are not ready yet: Limitations of state-of-the-art disease named entity recognizers10.1186/s13326-022-00280-6 ↗
34. Getoor, L., Friedman, N., Koller, D., Pfeffer, A., and Taskar, B. (2007). Probabilistic relational models. Introduction to Statistical Relational Learning, MIT Press.10.7551/mitpress/7432.001.0001 ↗
35. Zhang, X., Zhan, K., Hu, E., Fu, C., Luo, L., Jiang, H., Jia, Y., Yu, F., Dou, Z., and Cao, Z. (2021, January 11–15). Answer complex questions: Path ranker is all you need. Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual.10.1145/3404835.3462942 ↗
36. GPT-4 is here: What scientists think10.1038/d41586-023-00816-5 ↗
37. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). LLaMa 2: Open foundation and fine-tuned chat models. arXiv.
38. Simple and effective masked diffusion language models
39. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
40. Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., and Gallé, M. (2022). Bloom: A 176b-parameter open-access multilingual language model. arXiv.
41. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., and Lin, X.V. (2022). OPT: Open pre-trained transformer language models. arXiv.
42. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. (2023). The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. arXiv.
43. Language models are few-shot learners
44. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.D.L., Hendricks, L.A., Welbl, J., and Clark, A. (2022). Training compute-optimal large language models. arXiv.
45. PaLM: Scaling language modeling with pathways
46. Anthropic, P.B.C. (2026, January 17). Introducing Claude. Available online: https://www.anthropic.com/news/introducing-claude.
47. Training language models to follow instructions with human feedback
48. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
49. Direct preference optimization: Your language model is secretly a reward model
50. Yuan, W., Pang, R.Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J.E. (2024, January 21–27). Self-Rewarding Language Models. Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria. Available online: https://openreview.net/forum?id=0NphYCmgua.
51. Dong, G., Yuan, H., Lu, K., Li, C., Xue, M., Liu, D., Wang, W., Yuan, Z., Zhou, C., and Zhou, J. (2024, January 11–16). How abilities in large language models are affected by supervised fine-tuning data composition. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2024.acl-long.12 ↗
52. Rank analysis of incomplete block designs: I. the method of paired comparisons
53. Explainability for large language models: A survey
54. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models10.1162/COLI.a.16 ↗
55. Chen, J., Huang, X., and Li, Y. (2026, January 17). Dynamic supplementation of federated search results for reducing hallucinations in LLMs. Available online: https://osf.io/preprints/osf/x5vge_v1.
56. A global joint model for semantic role labeling10.1162/coli.2008.34.2.161 ↗
57. Johansson, R., and Nugues, P. (2008, January 16). Dependency-based syntactic–semantic analysis with PropBank and NomBank. Proceedings of the Conference on Computational Natural Language Learning, Manchester, UK. Available online: https://aclanthology.org/W08-2123.pdf.10.3115/1596324.1596355 ↗
58. Zhou, J., and Xu, W. (2015, January 26–31). End-to-end learning of semantic role labeling using recurrent neural networks. Proceedings of the Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China.10.3115/v1/P15-1109 ↗
59. Björkelund, A., Hafdell, L., and Nugues, P. (2009, January 4). Multilingual semantic role labeling. Proceedings of the Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, Boulder, CO, USA.10.3115/1596409.1596416 ↗
60. WordNet: A lexical database for English10.1145/219717.219748 ↗
61. Yago: A large ontology from Wikipedia and WordNet10.1016/j.websem.2008.06.001 ↗
62. Distributional semantics and linguistic theory10.1146/annurev-linguistics-011619-030303 ↗
63. Distributional models of word meaning10.1146/annurev-linguistics-030514-125254 ↗
64. The distributional hypothesis
65. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
66. Li, X.L., and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv.10.18653/v1/2021.acl-long.353 ↗
67. Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.P., Bing, L., Xu, X., Poria, S., and Lee, R. (2023, January 6–10). LLM-adapters: An adapter family for parameter-efficient fine-tuning of large language models. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.319 ↗
68. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset
69. Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S.Q. (2024). Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv.
70. LoRA: Low-rank adaptation of large language models
71. A review of recent work in transfer learning and domain adaptation for natural language processing of electronic health records10.1055/s-0041-1726522 ↗
72. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, H., and Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv.
73. Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. (2023). Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv.
74. Chain-of-thought prompting elicits reasoning in large language models
75. Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. (2022, January 25–29). Large language models are human-level prompt engineers. Proceedings of the International Conference on Learning Representations, Virtual.
76. Li, C., Liang, J., Zeng, A., Chen, X., Hausman, K., Sadigh, D., Levine, S., Fei-Fei, L., Xia, F., and Ichter, B. (2023). Chain of code: Reasoning with a language model-augmented code emulator. arXiv.
77. Shin, T., Razeghi, Y., Logan, R.L., Wallace, E., and Singh, S. (2020). Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv.10.18653/v1/2020.emnlp-main.346 ↗
78. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J.Z., and Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv.
79. Tian, J., He, Z., Dai, X., Ma, C.Y., Liu, Y.C., and Kira, Z. (2023, January 17–24). Trainable projected gradient method for robust fine-tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00757 ↗
80. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery
81. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., and Cao, Y. (2023, January 1–5). ReAct: Synergizing reasoning and acting in language models. Proceedings of the International Conference on Learning Representations, Kigali, Rwanda.
82. Masterman, T., Besen, S., Sawtell, M., and Chao, A. (2024). The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. arXiv.
83. Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L., Liu, Q., Zhou, M., and Zhang, Z. (2024). Do not think that much for 2+3=? on the overthinking of o1-like LLMs. arXiv.
84. Liu, N., Chen, L., Tian, X., Zou, W., Chen, K., and Cui, M. (2024). From LLM to conversational agent: A memory enhanced architecture with fine-tuning of large language models. arXiv.
85. Reflexion: Language agents with verbal reinforcement learning
86. Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance10.3354/cr030079 ↗
87. Yacouby, R., and Axman, D. (2020, January 20). Probabilistic extension of precision, recall, and f1 score for more thorough evaluation of classification models. Proceedings of the Workshop on Evaluation and Comparison of NLP Systems, Online.10.18653/v1/2020.eval4nlp-1.9 ↗
88. Chapelle, O., Metlzer, D., Zhang, Y., and Grinspan, P. (2009, January 2–6). Expected reciprocal rank for graded relevance. Proceedings of the ACM Conference on Information and Knowledge Management, Hong Kong, China.10.1145/1645953.1646033 ↗
89. A structured review of the validity of BLEU10.1162/coli_a_00322 ↗
90. Learning to summarize with human feedback
91. Wang, C., Deng, Y., Lyu, Z., Zeng, L., He, J., Yan, S., and An, B. (2024). Q*: Improving multi-step reasoning for LLMs with deliberative planning. arXiv.
92. Judging LLM-as-a-Judge with MT-Bench and chatbot arena
93. The meaning and use of the area under a receiver operating characteristic (ROC) curve10.1148/radiology.143.1.7063747 ↗
94. Text similarity measures in news articles by vector space model using NLP10.1007/s40031-020-00501-5 ↗
95. Wahba, Y., Madhavji, N., and Steinbacher, J. (2022). A comparison of SVM against pre-trained language models (PLMs) for text classification tasks. International Conference on Machine Learning, Optimization, and Data Science, Springer Nature.10.1007/978-3-031-25891-6_23 ↗
96. A maximum entropy approach to adaptive statistical language modelling10.1006/csla.1996.0011 ↗
97. Yin, J., and Wang, J. (2014, January 24–27). A Dirichlet multinomial mixture model-based approach for short text clustering. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA.10.1145/2623330.2623715 ↗
98. Yin, W., Kann, K., Yu, M., and Schütze, H. (2017). Comparative study of CNN and RNN for natural language processing. arXiv.
99. Searching for efficient transformers for language modeling
100. Generative adversarial networks: A literature review