Efficient Tree Policy with Attention-Based State Representation for Interactive Recommendation
Source abstract
Abstract
Nowadays, interactive recommendation systems (IRS) play a significant role in our daily life. Recently, reinforcement learning has shown great potential in solving challenging tasks in IRS, since it can focus on long-term profit and can capture the dynamic preference of users. However, existing RL methods for IRS have two typical deficiencies. First, most state representation models use left-to-right recurrent neural networks to capture the user dynamics, which usually fail to handle the long and noisy sequential data in real life. Second, an IRS always needs to handle millions of items, leading to a large discrete action space in RL settings, which has not been fully addressed by the inefficient existing works. To bridge these deficiencies, in this paper, we propose attention-based tree recommendation (ATRec), an efficient tree-structured policy with attention-based state representation for IRS. ATRec uses an attention-based state representation model to effectively capture the user’s dynamic preference hidden in the long and noisy sequence of behaviors. Moreover, to improve the learning efficiency, we propose an efficient tree-structured policy representation method, in which a complete tree is devised to represent the policy, and a novel parameter-sharing strategy is introduced. Extensive experiments are conducted on three real-world datasets and the results show the proposed ATRec obtains 42.3% improvement over some of the state of the arts methods in the hit rate and 21.4% improvement in the mean reciprocal rank of the top 30 ranked items. Additionally, the learning and decision efficiency can also be improved at an average of 35.5%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Recommender Systems and Techniques · Advanced Bandit Algorithms Research · Advanced Graph Neural Networks
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer and Computing Science, Hangzhou City University, Hangzhou 310015, China
DeepBlue Academy of Sciences, Shanghai 200336, China
Data Science Institute, University of Technology Sydney, Sydney, NSW 2007, Australia
College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
Funding
Natural Science Foundation of Zhejiang Province · Zhejiang Provincial Key Research and Development Program of China
References (52)
Show source-provided references
1. Zou, L., Xia, L., Gu, Y., Zhao, X., Liu, W., Huang, J.X., and Yin, D. (2020, January 25–30). Neural Interactive Collaborative Filtering. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual.10.1145/3397271.3401181 ↗
2. Zhao, X., Zhang, W., and Wang, J. (November, January 27). Interactive collaborative filtering. Proceedings of the 22nd ACM international conference on Information and Knowledge Management, San Francisco, CA, USA.
3. Neural Time-Aware Sequential Recommendation by Jointly Modeling Preference Dynamics and Explicit Feature Couplings10.1109/TNNLS.2021.3069058 ↗
4. State representation modeling for deep reinforcement learning based recommendation10.1016/j.knosys.2020.106170 ↗
5. Chen, H., Dai, X., Cai, H., Zhang, W., Wang, X., Tang, R., Zhang, Y., and Yu, Y. (February, January 27). Large-scale Interactive Recommendation with Tree-structured Policy Gradient. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
6. Zhao, X., Zhang, L., Ding, Z., Xia, L., Tang, J., and Yin, D. (2018, January 19–23). Recommendations with negative feedback via pairwise deep reinforcement learning. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219886 ↗
7. Wang, S., Hu, L., Wang, Y., Sheng, Q.Z., Orgun, M., and Cao, L. (2019, January 10–16). Modeling multi-purpose sessions for next-item recommendations via mixture-channel purpose routing networks. Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2019/523 ↗
8. The era of intelligent recommendation: Editorial on intelligent recommendation with advanced AI and learning10.1109/MIS.2020.3026430 ↗
9. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.
10. Yang, L., Zheng, Q., and Pan, G. (2021, January 2–9). Sample complexity of policy gradient finding second-order stationary points. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i12.17271 ↗
11. Yang, L., Zheng, G., Zhang, Y., Zheng, Q., Li, P., and Pan, G. (2021, January 2–9). On convergence of gradient expected sarsa (λ). Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.
12. Shi, J.C., Yu, Y., Da, Q., Chen, S.Y., and Zeng, A.X. (February, January 27). Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
13. Zou, L., Xia, L., Ding, Z., Song, J., Liu, W., and Yin, D. (2019, January 4–8). Reinforcement learning to optimize long-term user engagement in recommender systems. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.10.1145/3292500.3330668 ↗
14. Zhou, S., Dai, X., Chen, H., Zhang, W., Ren, K., Tang, R., He, X., and Yu, Y. (2020, January 25–30). Interactive recommender system via knowledge graph-enhanced reinforcement learning. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual.10.1145/3397271.3401174 ↗
15. A Three-Layered Mutually Reinforced Model for Personalized Citation Recommendation10.1109/TNNLS.2018.2817245 ↗
16. Zhao, X., Xia, L., Zhang, L., Ding, Z., Yin, D., and Tang, J. (2018, January 2). Deep reinforcement learning for page-wise recommendations. Proceedings of the 12th ACM Conference on Recommender Systems, Vancouver, BC, USA.10.1145/3240323.3240374 ↗
17. Reinforcement learning based recommender systems: A survey10.1145/3543846 ↗
18. Wang, S., Hu, L., Cao, L., Huang, X., Lian, D., and Liu, W. (2018, January 2–7). Attention-based transactional context embedding for next-item recommendation. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI 2018. Association for the Advancement of Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11851 ↗
19. Hierarchical attentive transaction embedding with intra-and inter-transaction dependencies for next-item recommendation10.1109/MIS.2020.2997362 ↗
20. Wang, N., Wang, S., Wang, Y., Sheng, Q.Z., and Orgun, M. (2020, January 20–24). Modelling local and global dependencies for next-item recommendations. Proceedings of the International Conference on Web Information Systems Engineering, Amsterdam, The Netherlands.10.1007/978-3-030-62008-0_20 ↗
21. Song, W., Wang, S., Wang, Y., and Wang, S. (October, January 27). Next-item recommendations in short sessions. Proceedings of the 15th ACM Conference on Recommender Systems, Amsterdam, The Netherlands.
22. Zhang, J., Hao, B., Chen, B., Li, C., Chen, H., and Sun, J. (February, January 27). Hierarchical reinforcement learning for course recommendation in MOOCs. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
23. Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv.
24. Zheng, G., Zhang, F., Zheng, Z., Xiang, Y., Yuan, N.J., Xie, X., and Li, Z. (2018, January 23–27). DRN: A deep reinforcement learning framework for news recommendation. Proceedings of the 2018 World Wide Web Conference, Lyon, France.10.1145/3178876.3185994 ↗
25. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
26. Wang, S., Hu, L., Wang, Y., Cao, L., Sheng, Q.Z., and Orgun, M. (2019, January 10–16). Sequential recommender systems: Challenges, progress and prospects. Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China.10.24963/ijcai.2019/883 ↗
27. Chen, X., Li, S., Li, H., Jiang, S., Qi, Y., and Song, L. (2019, January 9–15). Generative adversarial user model for reinforcement learning based recommendation system. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
28. Zou, L., Xia, L., Du, P., Zhang, Z., Bai, T., Liu, W., Nie, J.Y., and Yin, D. (2020, January 3–7). Pseudo Dyna-Q: A reinforcement learning framework for interactive recommendation. Proceedings of the 13th International Conference on Web Search and Data Mining, Houston, TX, USA.10.1145/3336191.3371801 ↗
29. Lei, T., Zhang, Y., and Artzi, Y. (May, January 30). Training rnns as fast as cnns. Proceedings of the ICLR 2018, Vancouver, BC, Canada.
30. Gao, R., Xia, H., Li, J., Liu, D., Chen, S., and Chun, G. (2019, January 8–11). DRCGR: Deep reinforcement learning framework incorporating CNN and GAN-based for interactive recommendation. Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM), Beijing, China.10.1109/ICDM.2019.00122 ↗
31. Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., and Jiang, P. (2019, January 3–7). BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. Proceedings of the CIKM ’19: 28th ACM International Conference on Information and Knowledge Management, New York, NY, USA.10.1145/3357384.3357895 ↗
32. A survey on session-based recommender systems
33. REDRL: A review-enhanced deep reinforcement learning model for interactive recommendation10.1016/j.eswa.2022.118926 ↗
34. Wang, S., Hu, L., Wang, Y., He, X., Sheng, Q.Z., Orgun, M.A., Cao, L., Ricci, F., and Yu, P.S. (2021, January 19–26). Graph learning based recommender systems: A review. Proceedings of the 30th International Joint Conference on Artificial Intelligence, Virtual.10.24963/ijcai.2021/630 ↗
35. Wang, X., Xu, Y., He, X., Cao, Y., Wang, M., and Chua, T.S. (2020, January 20–24). Reinforced negative sampling over knowledge graph for recommendation. Proceedings of the Web Conference 2020, Taipei, Taiwan.10.1145/3366423.3380098 ↗
36. Wang, P., Fan, Y., Xia, L., Zhao, W.X., Niu, S., and Huang, J. (2020, January 25–30). KERL: A knowledge-guided reinforcement learning model for sequential recommendation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual.10.1145/3397271.3401134 ↗
37. Chen, X., Huang, C., Yao, L., Wang, X., and Zhang, W. (2020, January 19–24). Knowledge-guided deep reinforcement learning for interactive recommendation. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9207010 ↗
38. Xian, Y., Fu, Z., Muthukrishnan, S., De Melo, G., and Zhang, Y. (2019, January 21–25). Reinforcement knowledge graph reasoning for explainable recommendation. Proceedings of the 42nd international ACM SIGIR Conference on Research and Development in Information Retrieval, Paris, France.10.1145/3331184.3331203 ↗
39. Reinforcement learning with factored states and actions
40. Pazis, J., and Parr, R. (July, January 28). Generalized value functions for large action sets. Proceedings of the ICML, Washington, DC, USA.
41. The arcade learning environment: An evaluation platform for general agents10.1613/jair.3912 ↗
42. Chandak, Y., Theocharous, G., Kostas, J., Jordan, S., and Thomas, P. (2019, January 9–15). Learning action representations for reinforcement learning. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
43. Deep reinforcement learning framework for category-based item recommendation10.1109/TCYB.2021.3089941 ↗
44. A scalable hierarchical distributed language model
45. Funk, S. (2023, June 09). Netflix Update: Try This at Home. Available online: http://sifter.org/simon/journal/20061211.html.
46. Simple statistical gradient-following algorithms for connectionist reinforcement learning10.1007/BF00992696 ↗
47. He, R., and McAuley, J. (2016, January 11–15). Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. Proceedings of the 25th International Conference on World Wide Web, Montreal, QC, Canada.10.1145/2872427.2883037 ↗
48. McAuley, J., Targett, C., Shi, Q., and Van Den Hengel, A. (2015, January 9–13). Image-based recommendations on styles and substitutes. Proceedings of the 38th international ACM SIGIR Conference on Research and Development in Information Retrieval, Santiego, Chila.10.1145/2766462.2767755 ↗
49. Wang, S., Xu, X., Zhang, X., Wang, Y., and Song, W. (2022, January 25–29). Veracity-aware and event-driven personalized news recommendation for fake news mitigation. Proceedings of the ACM Web Conference 2022, Lyon, France.10.1145/3485447.3512263 ↗
50. Wang, S., Zhang, X., Wang, Y., Liu, H., and Ricci, F. (2022). Trustworthy recommender systems. arXiv.
51. Tang, J., and Wang, K. (2018, January 5–9). Personalized top-n sequential recommendation via convolutional sequence embedding. Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, Marina Del Rey, CA, USA.10.1145/3159652.3159656 ↗
52. Muja, M., and Lowe, D. (2023, June 09). Fast Library for Approximate Nearest Neighbors (FLANN). Available online: git://github.com/mariusmuja/flann.git or http://www.cs.ubc.ca/research/flann.