Enhancing Task-Oriented Dialogue Modeling through Coreference-Enhanced Contrastive Pre-Training
Source abstract
Abstract
Pre-trained language models (PLMs) are proficient at understanding context in plain text but often struggle with the nuanced linguistics of task-oriented dialogues. The information exchanges in dialogues and the dynamic role-shifting of speakers contribute to complex coreference and interlinking phenomena across multi-turn interactions. To address these challenges, we propose Coreference-Enhanced Contrastive Pre-training (CECPT), an innovative pre-training framework specifically designed to enhance dialogue modeling. CECPT utilizes unsupervised dialogue datasets to capture both semantic richness and structural coherence. Our experimental results demonstrate that the CECPT model significantly outperforms established baselines in three critical applications: intent recognition, dialogue act prediction, and dialogue state tracking. These findings suggest that CECPT is more adept at following the information flow within dialogues and accurately linking statuses to their respective references.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Speech and dialogue systems · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
JIUTIAN Team, China Mobile Research Institute, Beijing 100053, China
Funding
Beijing Natural Science Foundation · National Key R&D Program of China · China Mobile Holistic Artificial Intelligence Major Project Funding
References (36)
Show source-provided references
1. Wang, H., Wang, L., Du, Y., Chen, L., Zhou, J., Wang, Y., and Wong, K.F. (2023). A survey of the evolution of language model-based dialogue systems. arXiv.
2. Yang, X., Sheng, X., and Gu, J. (2023, January 6–8). A Task-Oriented Multi-turn Dialogue Mechanism for the Smart Cockpit. Proceedings of the Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data, Wuhan, China.10.1007/978-981-97-2390-4_22 ↗
3. Leveraging Transformer-based Pretrained Language model for Task-oriented dialogue system
4. Yi, Z., Ouyang, J., Liu, Y., Liao, T., Xu, Z., and Shen, Y. (2024). A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems. arXiv.
5. Recent advances in deep learning based dialogue systems: A systematic survey10.1007/s10462-022-10248-8 ↗
6. Wu, H., Zhou, H., Lan, M., Wu, Y., and Zhang, Y. (2023, January 9–14). Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.325 ↗
7. Jayanthi, S.M., Embar, V., and Raghunathan, K. (2021). Evaluating Pretrained Transformer Models for Entity Linking in Task-Oriented Dialog. arXiv.
8. Mo, Y., Yoo, J., and Kang, S. (2023). Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems. Mathematics, 11.10.3390/math11143048 ↗
9. Yang, Y., Li, Y., and Quan, X. (2021, January 2–9). Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2. Proceedings of the AAAI Conference on Artificial Intelligence, Virtually.10.1609/aaai.v35i16.17674 ↗
10. Wu, C.S., Hoi, S., Socher, R., and Xiong, C. (2020). TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. arXiv.10.18653/v1/2020.emnlp-main.66 ↗
11. Yan, S., Song, S., Li, J., Meng, S., and Hu, G. (2023, January 19–25). TITAN: Task-oriented dialogues with mixed-initiative interactions. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2023/583 ↗
12. Zhang, Z., Shen, L., Zhao, Y., Chen, M., and He, X. (2023, January 9–14). Dialog-Post: Multi-Level Self-Supervised Objectives and Hierarchical Model for Dialogue Post-Training. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.564 ↗
13. Siro, C., Aliannejadi, M., and de Rijke, M. (2024). Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems. arXiv.10.18653/v1/2024.findings-naacl.80 ↗
14. Dialogue relation extraction with document-level heterogeneous graph attention networks10.1007/s12559-023-10110-1 ↗
15. Qiu, J., Chen, Q., Dong, Y., Zhang, J., Yang, H., Ding, M., Wang, K., and Tang, J. (2020, January 6–10). Gcc: Graph contrastive coding for graph neural network pre-training. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Virtual Event.10.1145/3394486.3403168 ↗
16. Zhang, J., Bui, T., Yoon, S., Chen, X., Liu, Z., Xia, C., Tran, Q.H., Chang, W., and Yu, P. (2021). Few-shot intent detection via contrastive pre-training and fine-tuning. arXiv.10.18653/v1/2021.emnlp-main.144 ↗
17. Bai, X., Chen, Y., Song, L., and Zhang, Y. (2021). Semantic representation for dialogue modeling. arXiv.10.18653/v1/2021.acl-long.342 ↗
18. Vulić, I., Su, P.H., Coope, S., Gerz, D., Budzianowski, P., Casanueva, I., Mrkšić, N., and Wen, T.H. (2021). ConvFiT: Conversational fine-tuning of pretrained language models. arXiv.10.18653/v1/2021.emnlp-main.88 ↗
19. Coco-lm: Correcting and contrasting text sequences for language model pretraining
20. Sun, F.Y., Hoffmann, J., Verma, V., and Tang, J. (2019). Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv.
21. Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018). How powerful are graph neural networks?. arXiv.
22. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
23. Li, J., Zhu, Q., Luo, L., Liden, L., Huang, K., Shayandeh, S., Liang, R., Peng, B., Zhang, Z., and Shukla, S. (2021, January 2–9). Multi-domain task completion dialog challenge ii at dstc9. Proceedings of the 35th AAAI Conference on Artificial Intelligence, Ninth Dialog System Technology Challenge Workshop, Virtually.
24. Rastogi, A., Zang, X., Sunkara, S., Gupta, R., and Khaitan, P. (2020, January 7–12). Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6394 ↗
25. Byrne, B., Krishnamoorthi, K., Sankar, C., Neelakantan, A., Duckworth, D., Yavuz, S., Goodrich, B., Dubey, A., Cedilnik, A., and Kim, K.Y. (2019). Taskmaster-1: Toward a realistic and diverse dialog dataset. arXiv.10.18653/v1/D19-1459 ↗
26. Budzianowski, P., Wen, T.H., Tseng, B.H., Casanueva, I., Ultes, S., Ramadan, O., and Gašić, M. (2018). MultiWOZ–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv.10.18653/v1/D18-1547 ↗
27. Li, X., Wang, Y., Sun, S., Panda, S., Liu, J., and Gao, J. (2018). Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems. arXiv.
28. Eric, M., and Manning, C.D. (2017). Key-value retrieval networks for task-oriented dialogue. arXiv.10.18653/v1/W17-5506 ↗
29. Asri, L.E., Schulz, H., Sharma, S., Zumer, J., Harris, J., Fine, E., Mehrotra, R., and Suleman, K. (2017). Frames: A corpus for adding memory to goal-oriented dialogue systems. arXiv.
30. Mrkšić, N., Séaghdha, D.O., Wen, T.H., Thomson, B., and Young, S. (2016). Neural belief tracker: Data-driven dialogue state tracking. arXiv.10.18653/v1/P17-1163 ↗
31. Wen, T.H., Vandyke, D., Mrksic, N., Gasic, M., Rojas-Barahona, L.M., Su, P.H., Ultes, S., and Young, S. (2016). A network-based end-to-end trainable task-oriented dialogue system. arXiv.10.18653/v1/E17-1042 ↗
32. Larson, S., Mahendran, A., Peper, J.J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J.K., Leach, K., Laurenzano, M.A., and Tang, L. (2019). An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. arXiv.10.18653/v1/D19-1131 ↗
33. Henderson, M., Thomson, B., and Williams, J.D. (2014, January 18–20). The second dialog state tracking challenge. Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), Philadelphia, PA, USA.10.3115/v1/W14-4337 ↗
34. Shah, P., Hakkani-Tür, D., Liu, B., and Tür, G. (2018, January 1–6). Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), New Orleans, LA, USA.10.18653/v1/N18-3006 ↗
35. Language models are unsupervised multitask learners
36. Zhang, Y., Sun, S., Galley, M., Chen, Y.C., Brockett, C., Gao, X., Gao, J., Liu, J., and Dolan, B. (2019). Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv.10.18653/v1/2020.acl-demos.30 ↗