Variational Reward Estimator Bottleneck: Towards Robust Reward Estimator for Multidomain Task-Oriented Dialogue
Source abstract
Abstract
Despite its significant effectiveness in adversarial training approaches to multidomain task-oriented dialogue systems, adversarial inverse reinforcement learning of the dialogue policy frequently fails to balance the performance of the reward estimator and policy generator. During the optimization process, the reward estimator frequently overwhelms the policy generator, resulting in excessively uninformative gradients. We propose the variational reward estimator bottleneck (VRB), which is a novel and effective regularization strategy that aims to constrain unproductive information flows between inputs and the reward estimator. The VRB focuses on capturing discriminative features by exploiting information bottleneck on mutual information. Quantitative analysis on a multidomain task-oriented dialogue dataset demonstrates that the VRB significantly outperforms previous studies.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Speech and dialogue systems · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Korea University, 145, Anam-ro, Seongbuk-gu, Seoul 02841, Korea
Funding
Ministry of Science and ICT, South Korea
References (40)
Show source-provided references
1. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., and Graepel, T. (2019). Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. CoRR.10.1038/s41586-020-03051-4 ↗
2. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
3. Deep Blue10.1016/S0004-3702(01)00129-1 ↗
4. A world championship caliber checkers program10.1016/0004-3702(92)90074-8 ↗
5. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals10.1126/science.aao1733 ↗
6. Moravcík, M., Schmid, M., Burch, N., Lisý, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowling, M.H. (2017). DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker. CoRR.10.1126/science.aam6960 ↗
7. Reinforcement Learning of Motor Skills with Policy Gradients10.1016/j.neunet.2008.02.003 ↗
8. Dy, J., and Krause, A. (2018, January 10–15). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.
9. Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A.A.A., Yogamani, S.K., and Pérez, P. (2020). Deep Reinforcement Learning for Autonomous Driving: A Survey. CoRR.
10. Wu, J., Huang, Z., and Lv, C. (2021). Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving. arXiv.
11. Zhao, X., Zhang, L., Ding, Z., Xia, L., Tang, J., and Yin, D. (2018, January 19–23). Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219886 ↗
12. Deep Learning Based Recommender System10.1145/3158369 ↗
13. Zhao, T., and Eskenazi, M. (2016). Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning. Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics.10.18653/v1/W16-3601 ↗
14. Dhingra, B., Li, L., Li, X., Gao, J., Chen, Y.N., Ahmed, F., and Deng, L. (August, January 30). Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, AB, Canada.
15. Shi, W., and Yu, Z. (2018, January 15–20). Sentiment Adaptive End-to-End Dialog Systems. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, QC, Australia.10.18653/v1/P18-1140 ↗
16. Shah, P., Hakkani-Tür, D., Liu, B., and Tür, G. (2018). Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), Association for Computational Linguistics.10.18653/v1/N18-3006 ↗
17. Ziebart, B.D., Maas, A.L., Bagnell, J.A., and Dey, A.K. (2008). Maximum Entropy Inverse Reinforcement Learning, AAAI.
18. Russell, S. (1998, January 24–26). Learning agents for uncertain environments. Proceedings of the Eleventh Annual Conference on Computational Learning Theory, Madison, WI, USA.10.1145/279943.279964 ↗
19. Ng, A., and Russell, S. (July, January 29). Algorithms for Inverse Reinforcement Learning. Proceedings of the ICML’00 Seventeenth International Conference on Machine Learning, Stanford, CA, USA.
20. Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., and Garnett, R. (2016). Generative Adversarial Imitation Learning. Advances in Neural Information Processing Systems 29, International Barcelona Convention Center, Curran Associates, Inc.
21. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems 27, Curran Associates, Inc.
22. Fu, J., Luo, K., and Levine, S. (May, January 30). Learning Robust Rewards with Adverserial Inverse Reinforcement Learning. Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada.
23. Takanobu, R., Zhu, H., and Huang, M. (2019). Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics.10.18653/v1/D19-1010 ↗
24. Tishby, N., Pereira, F.C., and Bialek, W. (1999, January 22–24). The information bottleneck method. Proceedings of the 37-th Annual Allerton Conference on Communication, Control and Computing, Monticello, IL, USA.
25. Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2016). Deep Variational Information Bottleneck. arXiv.
26. Peng, X.B., Kanazawa, A., Toyer, S., Abbeel, P., and Levine, S. (2019, January 6–9). Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
27. The Dialog State Tracking Challenge Series: A Review10.5087/dad.2016.301 ↗
28. Zhang, Z., Huang, M., Zhao, Z., Ji, F., Chen, H., and Zhu, X. (2019). Memory-Augmented Dialogue Management for Task-Oriented Dialogue Systems. ACM Trans. Inf. Syst., 37.10.1145/3317612 ↗
29. Wu, C.S., Madotto, A., Hosseini-Asl, E., Xiong, C., Socher, R., and Fung, P. (August, January 28). Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
30. Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., and Young, S. (2007). Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System. Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, Association for Computational Linguistics.10.3115/1614108.1614146 ↗
31. Gür, I., Hakkani-Tür, D., Tür, G., and Shah, P. (2018, January 18–21). User modeling for task oriented dialogues. Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.10.1109/SLT.2018.8639652 ↗
32. Trust Region Policy Optimization
33. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
34. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016, January 2–4). High-dimensional continuous control using generalized advantage estimation. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico.
35. Finn, C., Levine, S., and Abbeel, P. (2016, January 19–24). Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization. Proceedings of the 33rd International Conference on International Conference on Machine Learning—Volume 48. JMLR.org, ICML’16, New York, NY, USA.
36. Budzianowski, P., Wen, T.H., Tseng, B.H., Casanueva, I., Ultes, S., Ramadan, O., and Gašić, M. (2018). MultiWOZ—A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics.10.18653/v1/D18-1547 ↗
37. Gašić, M., Mrkšić, N., Su, P., Vandyke, D., Wen, T., and Young, S. (2015, January 13–17). Policy committee for adaptation in multi-domain spoken dialogue systems. Proceedings of the 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Scottsdale, AZ, USA.10.1109/ASRU.2015.7404871 ↗
38. Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., and de Freitas, N. (2017, January 24–26). Sample Efficient Actor-Critic with Experience Replay. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
39. Liu, B., and Lane, I. (2018). Adversarial Learning of Task-Oriented Neural Dialog Models. Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Association for Computational Linguistics.10.18653/v1/W18-5041 ↗
40. A Bayesian Committee Machine10.1162/089976600300014908 ↗