Time-Varying Preference Bandits for Robot Behavior Personalization
Source abstract
Abstract
Robots are increasingly employed in diverse services, from room cleaning to coffee preparation, necessitating an accurate understanding of user preferences. Traditional preference-based learning allows robots to learn these preferences through iterative queries about desired behaviors. However, these methods typically assume static human preferences. In this paper, we challenge this static assumption by considering the dynamic nature of human preferences and introduce the discounted preference bandit method to manage these changes. This algorithm adapts to evolving human preferences and supports seamless human–robot interaction through effective query selection. Our approach outperforms existing methods in time-varying scenarios across three key performance metrics.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Bandit Algorithms Research · Data Stream Mining Techniques · Reinforcement Learning in Robotics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence, Chung-Ang University, Seoul 06974, Republic of Korea
School of Computer Science and Engineering, Chung-Ang University, Seoul 06974, Republic of Korea
Department of Statistics, Korea University, Seoul 02841, Republic of Korea
Funding
Chung-Ang University Graduate Research Scholarship · Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)
References (25)
Show source-provided references
1. Sadigh, D., Sastry, S., Seshia, S.A., and Dragan, A.D. (2016, January 12–16). Planning for autonomous cars that leverage effects on human actions. Proceedings of the Robotics: Science and Systems, Ann Arbor, MI, USA.
2. Weiss, A., and Huber, A. (2016). User Experience of a Smart Factory Robot: Assembly Line Workers Demand Adaptive Robots. arXiv.
3. Wilde, N., Kulic, D., and Smith, S.L. (2020, January 25–29). Active Preference Learning using Maximum Regret. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341530 ↗
4. Sadigh, D., Dragan, A.D., Sastry, S., and Seshia, S.A. (2017, January 12–16). Active Preference-Based Learning of Reward Functions. Proceedings of the Robotics: Science and Systems XIII, Cambridge, MA, USA.10.15607/RSS.2017.XIII.053 ↗
5. Biyik, E., and Sadigh, D. (2018, January 29–31). Batch Active Preference-Based Learning of Reward Functions. Proceedings of the Annual Conference on Robot Learning, CoRL, Zürich, Switzerland.
6. Biyik, E., Palan, M., Landolfi, N.C., Losey, D.P., and Sadigh, D. (November, January 30). Asking Easy Questions: A User-Friendly Approach to Active Reward Learning. Proceedings of the Annual Conference on Robot Learning, CoRL, Osaka, Japan.
7. Biyik, E., Wang, K., Anari, N., and Sadigh, D. (2019). Batch Active Learning Using Determinantal Point Processes. arXiv.
8. Active preference-based Gaussian process regression for reward learning and optimization10.1177/02783649231208729 ↗
9. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences10.1177/02783649211041652 ↗
10. Franklin, M., Ashton, H., Gormann, R., and Armstrong, S. (2022). Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AI. arXiv.
11. Basu, C., Biyik, E., He, Z., Singhal, M., and Sadigh, D. (2019, January 3–8). Active Learning of Reward Dynamics from Hierarchical Queries. Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.10.1109/IROS40897.2019.8968522 ↗
12. Russac, Y., Vernade, C., and Cappé, O. (2019, January 8–14). Weighted Linear Bandits for Non-Stationary Environments. Proceedings of the Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
13. Lattimore, T., and Szepesvári, C. (2020). Bandit Algorithms, Cambridge University Press.10.1017/9781108571401 ↗
14. Chan, L., Hadfield-Menell, D., Srinivasa, S.S., and Dragan, A.D. (2019, January 11–14). The Assistive Multi-Armed Bandit. Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction, Daegu, Republic of Korea.10.1109/HRI.2019.8673234 ↗
15. Inverse Contextual Bandits: Learning How Behavior Evolves over Time
16. Tian, R., Tomizuka, M., Dragan, A.D., and Bajcsy, A. (2023, January 13–16). Towards Modeling and Influencing the Dynamics of Human Learning. Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Stockholm, Sweden.10.1145/3568162.3578629 ↗
17. Russac, Y., Cappé, O., and Garivier, A. (2020). Algorithms for Non-Stationary Generalized Linear Bandits. arXiv.
18. Faury, L., Russac, Y., Abeille, M., and Calauzènes, C. (2021). Regret Bounds for Generalized Linear Bandits under Parameter Drift. arXiv.
19. Chen, X., Li, Y., and Mao, J. (2018, January 7–10). A Nearly Instance Optimal Algorithm for Top-k Ranking under the Multinomial Logit Model. Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), New Orleans, LA, USA.10.1137/1.9781611975031.160 ↗
20. Finite-time analysis of the multiarmed bandit problem10.1023/A:1013689704352 ↗
21. Todorov, E., Erez, T., and Tassa, Y. (2012, January 7–12). MuJoCo: A physics engine for model-based control. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, Vilamoura-Algarve, Portugal.10.1109/IROS.2012.6386109 ↗
22. Bajcsy, A., Losey, D.P., O’Malley, M.K., and Dragan, A.D. (2017, January 13–15). Learning Robot Objectives from Physical Human Interaction. Proceedings of the Annual Conference on Robot Learning, CoRL, Mountain View, CA, USA.
23. Bauckhage, C. (2015). Numpy/SciPy Recipes for Data Science: k-Medoids Clustering, University of Bonn.
24. Kulesza, A., and Taskar, B. (July, January 28). k-DPPs: Fixed-Size Determinantal Point Processes. Proceedings of the International Conference on Machine Learning, ICML, Bellevue, WA, USA.
25. (2024, November 24). Doosan Robotics. A0912 Product Series, 2024, Available online: https://www.doosanrobotics.com/ko/products/series/a0912.