Using Large Language Models for Goal-Oriented Dialogue Systems
Source abstract
Abstract
In the development of goal-oriented dialogue systems, neural network topic modeling and clustering methods are traditionally used to extract user intentions and operator response scenario blocks. The emergence of generative large language models allows one to radically change the approach to generate dialogue scenarios in the form of a graph with context preservation. In this article we analyzed seven popular large language models on prepared test prompts for Russian and English languages for intent mining and named entity recognition. The present study aimed to investigate the effectiveness of two methods for constructing dialogues in goal-oriented dialogue systems: the heuristic-based approach with additional training on labeled data and the prompt-based approach without such training. The primary objective was to evaluate the impact of incorporating labeled dialogue data on the quality of constructed dialogues, with a focus on dialogue context. The study emphasized the need for dialogue systems to consider the dialogue context in constructing goal-oriented dialogues. The two approaches were compared for the MultiWOZ 2.2 and MANTiS dialogue corpora on a locally deployed LLaMA model. The results showed that the LlaMA model without training on labeled dialogues achieved a BERTScore metric value of 0.75 for the MultiWOZ dataset and 0.72 for the MANTiS dataset, and the LlaMA model with training on labeled dialogues achieved a BERTScore metric value of 0.85 for the MultiWOZ dataset and 0.82 for the MANTiS dataset. This finding has practical implications for the development of more effective dialogue systems in the field of customer service that can engage users in more productive and meaningful machine-to-human interactions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and dialogue systems · Topic Modeling · Multi-Agent Systems and Negotiation
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Research Institute of Digital Intelligent Technologies, Orenburg State University, Pobedy Pr. 13, Orenburg 460018, Russia
Funding
Russian Science Foundation
References (55)
Show source-provided references
1. Sood, P., Tanwar, H., Singh, J., Ruhela, A.K., Gupta, N., and Kumar, R. (2024, January 21–23). Revolutionizing Customer Service: An AI-powered Chatbot Approach using Advanced NLP Techniques. Proceedings of the 2024 3rd Edition of IEEE Delhi Section Flagship Conference (DELCON), New Delhi, India.10.1109/DELCON64804.2024.10866078 ↗
2. Rustamov, S., Bayramova, A., and Alasgarov, E. (2021). Development of dialogue management system for banking services. Appl. Sci., 11.10.3390/app112210995 ↗
3. An intelligent knowledge-based chatbot for customer service10.1016/j.elerap.2021.101098 ↗
4. Addlesee, A., Sieińska, W., Gunson, N., Garcia, D.H., Dondrup, C., and Lemon, O. (2023). Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineering. arXiv.10.18653/v1/2023.sigdial-1.22 ↗
5. Lahiri, A., Sanyal, D.K., and Mukherjee, I. (2023). CitePrompt: Using Prompts to Identify Citation Intent in Scientific Papers. arXiv.10.1109/JCDL57899.2023.00017 ↗
6. Nambanoor Kunnath, S., Pride, D., and Knoth, P. (2023, January 21–25). Prompting Strategies for Citation Classification. Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, Birmingham, UK.10.1145/3583780.3615018 ↗
7. Chang, K.W., Tseng, W.C., Li, S.W., and Lee, H.Y. (2022). SpeechPrompt: An exploration of prompt tuning on generative spoken language model for speech processing tasks. arXiv.10.21437/Interspeech.2022-10610 ↗
8. Dighe, P., Su, Y., Zheng, S., Liu, Y., Garg, V., Niu, X., and Tewfik, A. (2023). Leveraging Large Language Models for Exploiting ASR Uncertainty. arXiv.10.1109/ICASSP48485.2024.10446132 ↗
9. Gao, N., Zhao, Z., Zeng, Z., Zhang, S., Weng, D., and Bao, Y. (2023). GesGPT: Speech Gesture Synthesis with Text Parsing from GPT. arXiv.10.1109/LRA.2024.3359544 ↗
10. Zhang, R.H., Sell, P., Zhang, Y., Che, L., Gao, A., Sathiyajith, K.S., Bhatt, R., Nagasubramaniam, P., Vummanthala, S., and Dave, S. (2023). EvoquerBot: A Multimedia Chatbot Leveraging Synthetic Data for Cross-Domain Assistance. Penn State University.
11. Flex: Unifying evaluation for few-shot NLP
12. Loukas, L., Stogiannidis, I., Malakasiotis, P., and Vassos, S. (2023). Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance. arXiv.
13. Wang, P., He, K., Wang, Y., Song, X., Mou, Y., Wang, J., Xian, Y., Cai, X., and Xu, W. (2024). Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection. arXiv.
14. Abdullin, Y., Molla-Aliod, D., Ofoghi, B., Yearwood, J., and Li, Q. (2024). Synthetic Dialogue Dataset Generation using LLM Agents. arXiv.
15. Ahmed, R., Rauf, S.A., and Latif, S. (2024, January 19–20). Leveraging Large Language Models and Prompt Settings for Context-Aware Financial Sentiment Analysis. Proceedings of the 2024 5th International Conference on Advancements in Computational Sciences (ICACS), Lahore, Pakistan.10.1109/ICACS60934.2024.10473283 ↗
16. Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv.
17. Egger, R., and Yu, J. (2022). A Topic Modeling Comparison Between LDA, NMF, Top2Vec, and BERTopic to Demystify Twitter Posts. Front. Sociol., 7.10.3389/fsoc.2022.886498 ↗
18. A survey on evaluation of large language models10.1145/3641289 ↗
19. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., and Kumar, A. (2022). Holistic evaluation of language models. arXiv.
20. Benchmarking large language models for news summarization10.1162/tacl_a_00632 ↗
21. Explainability for large language models: A survey
22. Wang, Z. (2024, January 16). CausalBench: A Comprehensive Benchmark for Evaluating Causal Reasoning Capabilities of Large Language Models. Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10), Bangkok, Thailand.10.18653/v1/2024.sighan-1.17 ↗
23. Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., and Ma, L. (2023). Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv.
24. Wang, C., Liu, X., Yue, Y., Tang, X., Zhang, T., Jiayang, C., Yao, Y., Gao, W., Hu, X., and Qi, Z. (2023). Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv.
25. Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q., Li, Y., Gao, C., Huang, Y., Lyu, W., and Zhang, Y. (2024). Trustllm: Trustworthiness in large language models. arXiv.
26. McIntosh, T.R., Susnjak, T., Arachchilage, N., Liu, T., Watters, P., and Halgamuge, M.N. (2024). Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv.
27. Ilse, B., and Blackwood, F. (2024). Comparative analysis of finetuning strategies and automated evaluation metrics for large language models in customer service chatbots. Res. Sq., 1–18.10.21203/rs.3.rs-4895456/v1 ↗
28. Kaushal, A., Lin, C.C., Chauhan, R., and Kumar, R. (2024). Charting the Growth of Text Summarisation: A Data-Driven Exploration of Research Trends and Technological Advancements. Appl. Sci., 14.10.3390/app142311462 ↗
29. Falegnami, A., Tomassi, A., Corbelli, G., Nucci, F.S., and Romano, E. (2024). A Generative Artificial-Intelligence-Based Workbench to Test New Methodologies in Organisational Health and Safety. Appl. Sci., 14.10.3390/app142411586 ↗
30. Chen, Q., Zhou, W., Cheng, J., and Yang, J. (2024). An Enhanced Retrieval Scheme for a Large Language Model with a Joint Strategy of Probabilistic Relevance and Semantic Association in the Vertical Domain. Appl. Sci., 14.10.3390/app142411529 ↗
31. Vaškevičius, M., and Kapočiūtė-Dzikienė, J. (2024). Language Models for Predicting Organic Synthesis Procedures. Appl. Sci., 14.10.20944/preprints202411.1807.v1 ↗
32. Dai, Q., Mao, Y., Tang, J., and Rong, Y. (2024). STGPT2UGAN: Spatio-Temporal GPT-2 United Generative Adversarial Network for Wind Speed Prediction in Turbine Network. Appl. Sci., 14.10.3390/app142311217 ↗
33. Huang, J., Wang, M., Cui, Y., Liu, J., Chen, L., Wang, T., Li, H., and Wu, J. (2024). Layered Query Retrieval: An Adaptive Framework for Retrieval-Augmented Generation in Complex Question Answering for Large Language Models. Appl. Sci., 14.10.3390/app142311014 ↗
34. Bensch, C., Müller, A., Chojnowski, O., and Richert, A. (2024). Beyond Binary Dialogues: Research and Development of a Linguistically Nuanced Conversation Design for Social Robots in Group–Robot Interactions. Appl. Sci., 14.10.3390/app142210316 ↗
35. Smutny, P., and Bojko, M. (2024). Comparative Analysis of Chatbots Using Large Language Models for Web Development Tasks. Appl. Sci., 14.10.3390/app142110048 ↗
36. Benzinho, J., Ferreira, J., Batista, J., Pereira, L., Maximiano, M., Távora, V., Gomes, R., and Remédios, O. (2024). LLM Based Chatbot for Farm-to-Fork Blockchain Traceability Platform. Appl. Sci., 14.10.3390/app14198856 ↗
37. Liu, J., Tan, Y.K., Fu, B., and Lim, K.H. (2024). Intent-Aware Dialogue Generation and Multi-Task Contrastive Learning for Multi-Turn Intent Classification. arXiv.
38. Gao, J., Xiang, L., Wu, H., Zhao, H., Tong, Y., and He, Z. (2023). An Adaptive Prompt Generation Framework for Task-oriented Dialogue System. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics.10.18653/v1/2023.findings-emnlp.76 ↗
39. Cao, L. (2023). Diaggpt: An llm-based chatbot with automatic topic management for task-oriented dialogue. arXiv.
40. Hu, Z., Feng, Y., Deng, Y., Li, Z., Ng, S.K., Luu, A.T., and Hooi, B. (2023). Enhancing Large Language Model Induced Task-Oriented Dialogue Systems Through Look-Forward Motivated Goals. arXiv.
41. Wu, S., Shen, X., and Xia, R. (2023). A New Dialogue Response Generation Agent for Large Language Models by Asking Questions to Detect User’s Intentions. arXiv.
42. Stacey, J., Cheng, J., Torr, J., Guigue, T., Driesen, J., Coca, A., Gaynor, M., and Johannsen, A. (2024). LUCID: LLM-Generated Utterances for Complex and Interesting Dialogues. arXiv.10.18653/v1/2024.naacl-srw.8 ↗
43. Li, H., Yang, C., Zhang, A., Deng, Y., Wang, X., and Chua, T.S. (2024). Hello Again! LLM-powered Personalized Agent for Long-term Dialogue. arXiv.
44. Zhang, X., Peng, B., Li, K., Zhou, J., and Meng, H. (2023). Sgp-tod: Building task bots effortlessly via schema-guided llm prompting. arXiv.10.18653/v1/2023.findings-emnlp.891 ↗
45. Okadome, Y., Yuguchi, A., Fukui, R., and Matsumoto, Y. (2024). Prompt design using past dialogue summarization for llms to generate the current appropriate dialogue. International Conference on Artificial Neural Networks, Springer Nature.10.1007/978-3-031-72356-8_3 ↗
46. Robino, G. (2025). Conversation Routines: A Prompt Engineering Framework for Task-Oriented Dialog Systems. arXiv.
47. De Baer, J., Doğruöz, A.S., Demeester, T., and Develder, C. (2025). Single-vs. Dual-Prompt Dialogue Generation with LLMs for Job Interviews in Human Resources. arXiv.
48. Huang, Q., Liu, X., Ko, T., Wu, B., Wang, W., Zhang, Y., and Tang, L. (2024). Selective Prompting Tuning for Personalized Conversations with LLMs. arXiv.10.18653/v1/2024.findings-acl.959 ↗
49. Zang, X., Rastogi, A., Sunkara, S., Gupta, R., Zhang, J., and Chen, J. (2020). Multiwoz 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. arXiv.10.18653/v1/2020.nlp4convai-1.13 ↗
50. Penha, G., Balan, A., and Hauff, C. (2019). Introducing mantis: A novel multi-domain information seeking dialogues dataset. arXiv.
51. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv.
52. Lin, C.Y. (2004, January 25–26). Rouge: A package for automatic evaluation of summaries. Proceedings of the Text Summarization Branches Out, Barcelona, Spain.
53. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
54. Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
55. Retrieval-augmented generation for knowledge-intensive NLP tasks