Knowledge Interpolated Conditional Variational Auto-Encoder for Knowledge Grounded Dialogues
Source abstract
Abstract
In the Knowledge Grounded Dialogue (KGD) generation, the explicit modeling of instance-variety of knowledge specificity and its seamless fusion with the dialogue context remains challenging. This paper presents an innovative approach, the Knowledge Interpolated conditional Variational auto-encoder (KIV), to address these issues. In particular, KIV introduces a novel interpolation mechanism to fuse two latent variables: independently encoding dialogue context and grounded knowledge. This distinct fusion of context and knowledge in the semantic space enables the interpolated latent variable to guide the decoder toward generating more contextually rich and engaging responses. We further explore deterministic and probabilistic methodologies to ascertain the interpolation weight, capturing the level of knowledge specificity. Comprehensive empirical analysis conducted on the Wizard-of-Wikipedia and Holl-E datasets verifies that the responses generated by our model performs better than strong baselines, with notable performance improvements observed in both automatic metrics and manual evaluation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Speech and dialogue systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Konka Group Co., Ltd., Shenzhen 518053, China
Harbin Institute of Technology, Shenzhen 518055, China
Funding
National Natural Science Foundation of China · Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies · Natural Science Foundation of Guangdong · Key Technologies Research and Development Program of Shenzhen · HIT-KONKA Joint Lab
References (57)
Show source-provided references
1. Shang, L., Lu, Z., and Li, H. (2015, January 26–31). Neural Responding Machine for Short-Text Conversation. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, ACL 2015, Beijing, China.10.3115/v1/P15-1152 ↗
2. Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.Y., Gao, J., and Dolan, B. (June, January 31). A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Denver, CO, USA.10.3115/v1/N15-1020 ↗
3. Serban, I.V., Sordoni, A., Bengio, Y., Courville, A., and Pineau, J. (2016, January 12–17). Building end-to-end dialogue systems using generative hierarchical neural network models. Proceedings of the 30th AAAI Conference on Artificial Intelligence, AAAI 2016, Phoenix, AZ, USA.10.1609/aaai.v30i1.9883 ↗
4. Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., and Jurafsky, D. (2017, January 7–11). Adversarial Learning for Neural Dialogue Generation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark.10.18653/v1/D17-1230 ↗
5. Shen, L., and Feng, Y. (2020). CDL: Curriculum dual learning for emotion-controllable response generation. arXiv.10.18653/v1/2020.acl-main.52 ↗
6. Xu, J., Wang, H., Niu, Z., Wu, H., and Che, W. (2020, January 7–12). Knowledge graph grounded goal planning for open-domain conversation generation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6474 ↗
7. Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., and Weston, J. (2019, January 6–9). Wizard of Wikipedia: Knowledge-Powered Conversational Agents. Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA.
8. Qin, L., Galley, M., Brockett, C., Liu, X., Gao, X., Dolan, B., Choi, Y., and Gao, J. (August, January 28). Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy.10.18653/v1/P19-1539 ↗
9. Ghazvininejad, M., Brockett, C., Chang, M., Dolan, B., Gao, J., Yih, W., and Galley, M. (2018, January 2–7). A Knowledge-Grounded Neural Conversation Model. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, New Orleans, LA, USA.10.1609/aaai.v32i1.11977 ↗
10. Moghe, N., Arora, S., Banerjee, S., and Khapra, M.M. (November, January 31). Towards Exploiting Background Knowledge for Building Conversation Systems. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, Brussels, Belgium.10.18653/v1/D18-1255 ↗
11. Lian, R., Xie, M., Wang, F., Peng, J., and Wu, H. (2019). Learning to select knowledge for response generation in dialog systems. arXiv.10.24963/ijcai.2019/706 ↗
12. Xu, Y., Ishii, E., Winata, G.I., Lin, Z., Madotto, A., Liu, Z., Xu, P., and Fung, P. (2021, January 5). CAiRE in DialDoc21: Data augmentation for information seeking dialogue system. Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021), Online.10.18653/v1/2021.dialdoc-1.6 ↗
13. Yang, C., Lin, Z., Li, J., Meng, F., Wang, W., Wang, L., and Zhou, J. (2022, January 12–17). TAKE: Topic-shift aware knowledge selection for dialogue generation. Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea.
14. Komeili, M., Shuster, K., and Weston, J. (2021). Internet-augmented dialogue generation. arXiv.10.18653/v1/2022.acl-long.579 ↗
15. Varshney, D., Prabhakar, A., and Ekbal, A. (2022). Commonsense and named entity aware knowledge grounded dialogue generation. arXiv.10.18653/v1/2022.naacl-main.95 ↗
16. Yang, Y., Gao, Y., Li, J., and Huang, H. (2022). G2: Enhance Knowledge Grounded Dialogue via Ground Graph. arXiv.
17. Sun, Q., Xu, C., Hu, H., Wang, Y., Miao, J., Geng, X., Chen, Y., Xu, F., and Jiang, D. (2022). Stylized knowledge-grounded dialogue generation via disentangled template rewriting. arXiv.10.18653/v1/2022.naacl-main.241 ↗
18. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016, January 1–5). SQuAD: 100,000+ Questions for Machine Comprehension of Text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, TX, USA.10.18653/v1/D16-1264 ↗
19. Chen, D., Fisch, A., Weston, J., and Bordes, A. (August, January 30). Reading Wikipedia to Answer Open-Domain Questions. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, BC, Canada.10.18653/v1/P17-1171 ↗
20. Seo, M., Kembhavi, A., Farhadi, A., and Hajishirzi, H. (2016, January 2–4). Bidirectional attention flow for machine comprehension. Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, PR, USA.
21. Liu, S., Chen, H., Ren, Z., Feng, Y., Liu, Q., and Yin, D. (2018, January 15–20). Knowledge diffusion for neural dialogue generation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia.10.18653/v1/P18-1138 ↗
22. Ren, P., Chen, Z., Monz, C., Ma, J., and de Rijke, M. (February, January 27). Thinking Globally, Acting Locally: Distantly Supervised Global-to-Local Knowledge Selection for Background Based Conversation. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, HI, USA.
23. Meng, C., Ren, P., Chen, Z., Monz, C., Ma, J., and de Rijke, M. (February, January 27). RefNet: A reference-aware network for background based conversation. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, HI, USA.
24. Li, Y., Zhao, J., Lyu, M.R., and Wang, L. (2022). Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation. arXiv.10.18653/v1/2022.emnlp-main.721 ↗
25. Kim, B., Ahn, J., and Kim, G. (2020). Sequential latent knowledge selection for knowledge-grounded dialogue. arXiv.
26. Lin, X., Jian, W., He, J., Wang, T., and Chu, W. (2020, January 6–8). Generating informative conversational response using recurrent knowledge-interaction and knowledge-copy. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.6 ↗
27. Kingma, D.P., Mohamed, S., Rezende, D.J., and Welling, M. (2014, January 8–13). Semi-supervised learning with deep generative models. Proceedings of the 28th Conference on Neural Information Processing Systems, NIPS 2014, Montreal, QC, Canada.
28. Zhao, T., Zhao, R., and Eskenazi, M. (August, January 30). Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, BC, Canada.10.18653/v1/P17-1061 ↗
29. Song, H., Zhang, W., Cui, Y., Wang, D., and Liu, T. (2019, January 10–16). Exploiting Persona Information for Diverse Generation of Conversational Responses. Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China.10.24963/ijcai.2019/721 ↗
30. Massiceti, D., Siddharth, N., Dokania, P.K., and Torr, P.H. (2018, January 18–22). FlipDial: A generative model for two-way visual dialogue. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA.
31. Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019, January 10–15). Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, CA, USA.
32. Ye, H., Lo, K., Su, S., and Chen, Y. (February, January 27). Knowledge-Grounded Response Generation with Deep Attentional Latent-Variable Model. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, HI, USA.
33. Zheng, W., and Zhou, K. (2019, January 3–7). Enhancing Conversational Dialogue Models with Grounded Knowledge. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China.10.1145/3357384.3357889 ↗
34. Sun, W., Ren, P., and Ren, Z. (2023). Generative Knowledge Selection for Knowledge-Grounded Dialogues. arXiv.10.18653/v1/2023.findings-eacl.155 ↗
35. Prabhumoye, S., Hashimoto, K., Zhou, Y., Black, A.W., and Salakhutdinov, R. (2021). Focused attention improves document-grounded generation. arXiv.10.18653/v1/2021.naacl-main.338 ↗
36. Chen, Q., Lin, J., Zhang, Y., Ding, M., Cen, Y., Yang, H., and Tang, J. (2019). Towards knowledge-based recommender dialog system. arXiv.10.18653/v1/D19-1189 ↗
37. Wang, L., Hu, H., Sha, L., Xu, C., Wong, K.F., and Jiang, D. (2021). Finetuning large-scale pre-trained language models for conversational recommendation with knowledge graph. arXiv.
38. Zhou, K., Zhao, W.X., Bian, S., Zhou, Y., Wen, J.R., and Yu, J. (2020, January 6–10). Improving conversational recommender systems via knowledge graph based semantic fusion. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Virtual Event, USA.10.1145/3394486.3403143 ↗
39. Li, S., Namazifar, M., Jin, D., Bansal, M., Ji, H., Liu, Y., and Hakkani-Tur, D. (2022). Enhanced knowledge selection for grounded dialogues via document semantic graphs. arXiv.10.18653/v1/2022.naacl-main.202 ↗
40. Li, Z., Niu, C., Meng, F., Feng, Y., Li, Q., and Zhou, J. (August, January 28). Incremental Transformer with Deliberation Decoder for Document Grounded Conversations. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy.10.18653/v1/P19-1002 ↗
41. Wu, Z., Lu, B.R., Hajishirzi, H., and Ostendorf, M. (2021). DIALKI: Knowledge Identification in Conversational Systems through Dialogue-Document Contextualization. arXiv.10.18653/v1/2021.emnlp-main.140 ↗
42. Mi, L., He, T., Park, C.F., Wang, H., Wang, Y., and Shavit, N. (2021). Revisiting Latent-Space Interpolation via a Quantitative Evaluation Framework. arXiv.
43. Representation learning: A review and new perspectives10.1109/TPAMI.2013.50 ↗
44. Qian, S., Li, G., Cao, W.M., Liu, C., Wu, S., and Wong, H.S. (2019, January 10–16). Improving representation learning in autoencoders via multidimensional interpolation and dual regularizations. Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China.10.24963/ijcai.2019/453 ↗
45. Verma, V., Lamb, A., Kannala, J., Bengio, Y., and Lopez-Paz, D. (2019, January 10–16). Interpolation Consistency Training for Semi-supervised Learning. Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China.10.24963/ijcai.2019/504 ↗
46. Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., and Bengio, S. (2016, January 11–12). Generating Sentences from a Continuous Space. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany.10.18653/v1/K16-1002 ↗
47. Zhao, J.J., Kim, Y., Zhang, K., Rush, A.M., and LeCun, Y. (2018, January 10–15). Adversarially Regularized Autoencoders. Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden.
48. Gao, X., Lee, S., Zhang, Y., Brockett, C., Galley, M., Gao, J., and Dolan, B. (2019, January 2–7). Jointly Optimizing Diversity and Relevance in Neural Response Generation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2019, Minneapolis, MN, USA.10.18653/v1/N19-1125 ↗
49. Ganea, O., Gelly, S., Bécigneul, G., and Severyn, A. (2019, January 10–15). Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearities. Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, CA, USA.
50. Srivastava, A., and Sutton, C.A. (2017, January 24–26). Autoencoding Variational Inference For Topic Models. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
51. See, A., Liu, P.J., and Manning, C.D. (2017). Get to the point: Summarization with pointer-generator networks. arXiv.10.18653/v1/P17-1099 ↗
52. Liu, C.W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., and Pineau, J. (2016, January 1–5). How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1230 ↗
53. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. (2016, January 12–17). A Diversity-Promoting Objective Function for Neural Conversation Models. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2016, San Diego, CA, USA.10.18653/v1/N16-1014 ↗
54. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv.
55. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Sardinia, Italy.
56. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
57. Kullback, S. (1997). Information Theory and Statistics, Courier Corporation.