Multi-Turn Chatbot Based on Query-Context Attentions and Dual Wasserstein Generative Adversarial Networks
Source abstract
Abstract
To generate proper responses to user queries, multi-turn chatbot models should selectively consider dialogue histories. However, previous chatbot models have simply concatenated or averaged vector representations of all previous utterances without considering contextual importance. To mitigate this problem, we propose a multi-turn chatbot model in which previous utterances participate in response generation using different weights. The proposed model calculates the contextual importance of previous utterances by using an attention mechanism. In addition, we propose a training method that uses two types of Wasserstein generative adversarial networks to improve the quality of responses. In experiments with the DailyDialog dataset, the proposed model outperformed the previous state-of-the-art models based on various performance measures.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Program of Computer and Communications Engineering, Kangwon National University, Chuncheon 24341, Korea
Electronics and Telecommunications Research Institute, Daejeon 34129, Korea
Funding
Ministry of Science, ICT and Future Planning · Hyundai Motor Group
References (29)
Show source-provided references
1. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems, Curran Associates, Inc.
2. Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., and Jurafsky, D. (2017). Adversarial Learning for Neural Dialogue Generation. arXiv.10.18653/v1/D17-1230 ↗
3. Xu, Z., Liu, B., Wang, B., Chengjie, S.U., Wang, X., Wang, Z., and Qi, C. (2017, January 9). Neural Response Generation via GAN with an Approximate Embedding Layer. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1065 ↗
4. Yu, L., Zhang, W., Wang, J., and Yu, Y. (2017, January 13). Seqgan: Sequence Generative Adversarial Nets with Policy Gradient. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10804 ↗
5. Vinyals, O., and Le, Q. (2015). A neural conversational model. arXiv.
6. Shang, L., Lu, Z., and Li, H. (2015, January 26–31). Neural Responding Machine for Short-Text Conversation. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China.10.3115/v1/P15-1152 ↗
7. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. (2016, January 1–26). A Diversity-Promoting Objective Function for Neural Conversation Models. Proceedings of the NAACL-HLT, San Diego, CA, USA.10.18653/v1/N16-1014 ↗
8. Sato, S., Yoshinaga, N., Toyoda, M., and Kitsuregawa, M. (August, January 30). Modeling Situations in Neural Chatbots. Proceedings of the ACL 2017, Student Research Workshop, Vancouver, BC, Canada.10.18653/v1/P17-3020 ↗
9. Zhao, T., Zhao, R., and Eskenazi, M. (August, January 30). Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada.
10. Shen, X., Su, H., Niu, S., and Demberg, V. (2018, January 23–27). Improving Variational Encoder-Decoders in Dialogue Generation. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, Lyon, France.10.1609/aaai.v32i1.11960 ↗
11. Goyal, P., Hu, Z., Liang, X., Wang, C., and Xing, E.P. (2017, January 19–21). Nonparametric Variational Auto-Encoders for Hierarchical Representation Learning. Proceedings of the IEEE International Conference on Computer Vision, Sinaia, Romania.10.1109/ICCV.2017.545 ↗
12. Gu, X., Cho, K., Ha, J.W., and Kim, S. (2018). DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder. arXiv.
13. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Style Transfer from Non-Parallel Text by Cross-Alignment. Advances in Neural Information Processing Systems, Curran Associates, Inc.
14. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, Curran Associates, Inc.
15. Cho, K., Van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. arXiv.10.3115/v1/W14-4012 ↗
16. Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein GAN. arXiv.
17. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2–4). Tensorflow: A System for Large-Scale Machine Learning. Proceedings of the 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), Savannah, GA, USA.
18. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014, January 25–29). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1179 ↗
19. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified Linear Units Improve Restricted Boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel.
20. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
21. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Improved Training of Wasserstein GANs. Advances in Neural Information Processing Systems, Curran Associates, Inc.
22. Li, Y., Su, H., Shen, X., Li, W., Cao, Z., and Niu, S. (December, January 27). DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. Proceedings of the Eighth International Joint Conference on Natural Language Processing, Taipei, Taiwan.
23. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
24. Chen, B., and Cherry, C. (2014, January 26–27). A Systematic Comparison of Smoothing Techniques for Sentence-Level Bleu. Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, MD, USA.10.3115/v1/W14-3346 ↗
25. Liu, C.W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., and Pineau, J. (2016, January 1–4). How not to Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1230 ↗
26. Rus, V., and Lintean, M. (2012, January 7). A Comparison of Greedy and Optimal Assessment of Natural Language Student Input using Word-to-Word Similarity Metrics. Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, Montréal, QC, Canada.10.1007/978-3-642-30950-2_116 ↗
27. Mitchell, J., and Lapata, M. (2008, January 15–20). Vector-Based Models of Semantic Composition. Proceedings of the ACL-08: HLT, Columbus, OH, USA.
28. Forgues, G., Pineau, J., Larchevêque, J.M., and Tremblay, R. (2014, January 12). Bootstrapping Dialog Systems with Word Embeddings. Proceedings of the Nips, Modern Machine Learning and Natural Language Processing Workshop, Geneva, Switzerland.
29. Serban, I.V., Sordoni, A., Bengio, Y., Courville, A., and Pineau, J. (2016, January 12–17). Building End-to-End Dialogue Systems Using Generative Hierarchical Neural Network Models. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.9883 ↗