AraConv: Developing an Arabic Task-Oriented Dialogue System Using Multi-Lingual Transformer Model mT5
Source abstract
Abstract
Task-oriented dialogue systems (DS) are designed to help users perform daily activities using natural language. Task-oriented DS for English language have demonstrated promising performance outcomes; however, developing such systems to support Arabic remains a challenge. This challenge is mainly due to the lack of Arabic dialogue datasets. This study introduces the first Arabic end-to-end generative model for task-oriented DS (AraConv), which uses the multi-lingual transformer model mT5 with different settings. We also present an Arabic dialogue dataset (Arabic-TOD) and used it to train and test the proposed AraConv model. The results obtained are reasonable compared to those reported in the studies of English and Chinese using the same mono-lingual settings. To avoid problems associated with a small training dataset and to improve the AraConv model’s results, we suggest joint-training, in which the model is jointly trained on Arabic dialogue data and data from one or two high-resource languages such as English and Chinese. The findings indicate the AraConv model performed better in the joint-training setting than in the mono-lingual setting. The results obtained from AraConv on the Arabic dialogue dataset provide a baseline for other researchers to build robust end-to-end Arabic task-oriented DS that can engage with complex scenarios.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Speech and dialogue systems · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Technology, College of Computer and Information Sciences, King Saud University, P.O. Box 145111, Riyadh 4545, Saudi Arabia
References (66)
Show source-provided references
1. Lin, Z., Madotto, A., Winata, G.I., Xu, P., Jiang, F., Hu, Y., Shi, C., and Fung, P. (2021). BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling. arXiv.
2. Challenges in building intelligent open-domain dialog systems10.1145/3233771 ↗
3. Challenges facing the development of the Arabic chatbot10.1117/12.2240849 ↗
4. A Panoramic Survey of Natural Language Processing in the Arab World10.1145/3447735 ↗
5. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou’, R., Siddhant, A., Barua, A., and Raffel, C. (2020, January 15–20). mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, online.10.18653/v1/2021.naacl-main.41 ↗
6. McTear, M. (2020). Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots, Morgan & Claypool Publishers LLC.10.1007/978-3-031-02176-3 ↗
7. Qin, L., Xu, X., Che, W., Zhang, Y., and Liu, T. (2020, January 5–10). Dynamic Fusion Network for Multi-Domain End-to-end Task-Oriented Dialog. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, online.10.18653/v1/2020.acl-main.565 ↗
8. Lei, W., Jin, X., Ren, Z., He, X., Kan, M.Y., and Yin, D. (2018, January 15–20). Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.10.18653/v1/P18-1133 ↗
9. Budzianowski, P., and Vulić, I. (2019, January 4). Hello, It’s GPT-2-How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems. Proceedings of the 3rd Workshop on Neural Generation and Translation (WNGT 2019), Hong Kong, China.10.18653/v1/D19-5602 ↗
10. End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-210.18653/v1/2020.acl-main.54 ↗
11. A simple language model for task-oriented dialogue
12. SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching10.1162/tacl_a_00399 ↗
13. Yang, Y., Li, Y., and Quan, X. (2020). UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2. arXiv.10.1609/aaai.v35i16.17674 ↗
14. Lin, Z., Madotto, A., Winata, G.I., and Fung, P. (2020, January 16–20). MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), online.10.18653/v1/2020.emnlp-main.273 ↗
15. Wang, W., Zhang, Z., Guo, J., Dai, Y., Chen, B., and Luo, W. (2021). Task-Oriented Dialogue System as Natural Language Generation. arXiv.10.1145/3477495.3531920 ↗
16. Peng, B., Zhu, C., Li, C., Li, X., Li, J., Zeng, M., and Gao, J. (2020). Few-shot Natural Language Generation for Task-Oriented Dialog. arXiv.10.18653/v1/2020.findings-emnlp.17 ↗
17. Wu, C.-S., Hoi, S.C.H., Socher, R., and Xiong, C. (2020, January 16–20). TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), online.10.18653/v1/2020.emnlp-main.66 ↗
18. Madotto, A., Liu, Z., Lin, Z., and Fung, P. (2020). Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems. arXiv.10.18653/v1/2020.findings-emnlp.215 ↗
19. Campagna, G., Foryciarz, A., Moradshahi, M., and Lam, M. (2020). Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Tracking. arXiv.10.18653/v1/2020.acl-main.12 ↗
20. Zhang, Y., Ou, Z., Hu, M., and Feng, J. (2020, January 16–20). A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learning. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), online.10.18653/v1/2020.emnlp-main.740 ↗
21. Kulhánek, J., Hudeček, V., Nekvinda, T., and Dušek, O. (2021). AuGPT: Dialogue with Pre-trained Language Models and Data Augmentation. arXiv.
22. Gao, S., Takanobu, R., Peng, W., Liu, Q., and Huang, M. (2021). HyKnow: End-to-End Task-Oriented Dialog Modeling with Hybrid Knowledge Management. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Association for Computational Linguistics.10.18653/v1/2021.findings-acl.139 ↗
23. Lee, H., Lee, J., and Kim, T.Y. (2019). SUMBT: Slot-utterance matching for universal and scalable belief tracking. arXiv.10.18653/v1/P19-1546 ↗
24. BERT-DST: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer
25. Kim, S., Yang, S., Kim, G., and Lee, S.-W. (2020). Efficient Dialogue State Tracking by Selectively Overwriting Memory. arXiv.10.18653/v1/2020.acl-main.53 ↗
26. MA-DST: Multi-Attention-Based Scalable Dialog State Tracking
27. Heck, M., van Niekerk, C., Lubis, N., Geishauser, C., Lin, H.-C., Moresi, M., and Gašić, M. (2020). TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking. arXiv.10.18653/v1/2020.sigdial-1.4 ↗
28. Li, S., Yavuz, S., Hashimoto, K., Li, J., Niu, T., Rajani, N., Yan, X., Zhou, Y., and Xiong, C. (2020). CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers. arXiv.
29. Wang, D., Lin, C., Liu, Q., and Wong, K.-F. (2021, January 6–11). Fast and Scalable Dialogue State Tracking with Explicit Modular Decomposition. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Mexico City, Mexico.10.18653/v1/2021.naacl-main.27 ↗
30. Schuster, S., Shah, R., Gupta, S., and Lewis, M. (2019, January 2–7). Cross-lingual transfer learning for multilingual task oriented dialog. Proceedings of the NAACL-HLT 2019, Minneapolis, MN, USA.10.18653/v1/N19-1380 ↗
31. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems
32. Task-oriented dialog systems that consider multiple appropriate responses under the same context
33. Wang, K., Tian, J., Wang, R., Quan, X., and Yu, J. (2020). Multi-Domain Dialogue Acts and Response Co-Generation. arXiv.10.18653/v1/2020.acl-main.638 ↗
34. Implementation of A Neural Natural Language Understanding Component for Arabic Dialogue Systems10.1016/j.procs.2018.10.479 ↗
35. Elmadany, A.R.A., Abdou, S.M., and Gheith, M. (2018, January 7–12). Improving dialogue act classification for spontaneous Arabic speech and instant messages at utterance level. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Miyazaki, Japan.
36. Joukhadar, A., Saghergy, H., Kweider, L., and Ghneim, N. (2019, January 11–12). Arabic Dialogue Act Recognition for Textual Chatbot Systems. Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019), Trento, Italy.
37. Building an Arabic Flight Booking Dialogue System Using a Hybrid Rule-Based and Data Driven Approach10.1109/ACCESS.2021.3049732 ↗
38. Hijjawi, M., Bandar, Z., Crockett, K., and McLean, D. (2014, January 26). ArabChat: An arabic conversational agent. Proceedings of the 2014 6th International Conference on Computer Science and Information Technology, CSIT 2014-Proceedings, Amman, Jordan.10.1109/CSIT.2014.6806005 ↗
39. LABEEB: Intelligent Conversational Agent Approach to Enhance Course Teaching and Allied Learning Outcomes attainment10.4316/JACSM.201901001 ↗
40. LANA-I: An Arabic Conversational Intelligent Tutoring System for Children with ASD10.1007/978-3-030-22871-2_34 ↗
41. Arabic dialogue system for hotel reservation based on natural language processing techniques
42. Bendjamaa, F., and Nora, T. (2020, January 3). A Dialogue-System Using a Qur’anic Ontology. Proceedings of the 2020 Second International Conference on Embedded & Distributed Systems (EDiS), Oran, Algeria.10.1109/EDiS49545.2020.9296437 ↗
43. Fadhil, A., and AbuRa’Ed, A. (2019, January 2–4). Ollobot-Towards a text-based Arabic health conversational agent: Evaluation and results. Proceedings of the Recent Advances in Natural Language Processing (RANLP), Varna, Bulgaria.10.26615/978-954-452-056-4_034 ↗
44. Nabiha: An Arabic dialect chatbot
45. Attention is all you need
46. Mozannar, H., Maamary, E., el Hajal, K., and Hajj, H. (2019). Neural Arabic Question Answering. arXiv.10.18653/v1/W19-4612 ↗
47. Deep learning based question answering system in Bengali
48. Naous, T., Hokayem, C., and Hajj, H. (2020, January 8). Empathy-driven Arabic Conversational Chatbot. Proceedings of the Fifth Arabic Natural Language Processing Workshop, Barcelona, Spain.
49. Razumovskaia, E., Glavaš, G., Majewska, O., Ponti, E.M., Korhonen, A., and Vulić, I. (2021). Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems. arXiv.10.18653/v1/2022.acl-tutorials.8 ↗
50. He, X., Deng, L., Hakkani-Tur, D., and Tur, G. (2013, January 26–31). Multi-style adaptive training for robust cross-lingual spoken language understanding. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639292 ↗
51. Semantic Specialization of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints10.1162/tacl_a_00063 ↗
52. Castellucci, G., Bellomaria, V., Favalli, A., and Romagnoli, R. (2019). Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model. arXiv.
53. Bellomaria, V., Castellucci, G., Favalli, A., and Romagnoli, R. (2019). Almawave-SLU: A new dataset for SLU in Italian. arXiv.
54. Liu, Z., Shin, J., Xu, Y., Winata, G.I., Xu, P., Madotto, A., and Fung, P. (2019, January 11). Zero-shot cross-lingual dialogue systems with transferable latent variables. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1129 ↗
55. Dao, M.H., Truong, T.H., and Nguyen, D.Q. (2021). Intent Detection and Slot Filling for Vietnamese. arXiv.10.21437/Interspeech.2021-618 ↗
56. Survey on evaluation methods for dialogue systems10.1007/s10462-020-09866-x ↗
57. Exploring the limits of transfer learning with a unified text-to-text transformer
58. (2021, November 17). Huggingface/Transformers? Transformers: State-of-the-Art Natural Language Processing for Pytorch, TensorFlow, and JAX. Available online: https://github.com/huggingface/transformers.
59. (2021, November 17). PyTorch. Available online: https://pytorch.org/.
60. Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv.
61. Siblini, W., Pasqual, C., Lavielle, A., Challal, M., and Cauchois, C. (2019). Multilingual Question Answering from Formatted Text applied to Conversational Agents. arXiv.
62. Hsu, T.Y., Liu, C.L., and Lee, H.Y. (2019, January 3–7). Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1607 ↗
63. Upadhyay, S., Faruqui, M., Tür, G., Dilek, H.-T., and Heck, L. (2018, January 15–20). (Almost) Zero-shot cross-lingual spoken language understanding. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461905 ↗
64. Measuring nominal scale agreement among many raters10.1037/h0031619 ↗
65. Multilingual Denoising Pre-training for Neural Machine Translation10.1162/tacl_a_00343 ↗
66. Nagoudi, E.M.B., Elmadany, A., and Abdul-Mageed, M. (2021). AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation. arXiv.10.18653/v1/2022.acl-long.47 ↗