Causal Reinforcement Learning for Knowledge Graph Reasoning
Source abstract
Abstract
Knowledge graph reasoning can deduce new facts and relationships, which is an important research direction of knowledge graphs. Most of the existing methods are based on end-to-end reasoning which cannot effectively use the knowledge graph, so consequently the performance of the method still needs to be improved. Therefore, we combine causal inference with reinforcement learning and propose a new framework for knowledge graph reasoning. By combining the counterfactual method in causal inference, our method can obtain more information as prior knowledge and integrate it into the control strategy in the reinforcement model. The proposed method mainly includes the steps of relationship importance identification, reinforcement learning framework design, policy network design, and the training and testing of the causal reinforcement learning model. Specifically, a prior knowledge table is first constructed to indicate which relationship is more important for the problem to be queried; secondly, designing state space, optimization, action space, state transition and reward, respectively, is undertaken; then, the standard value is set and compared with the weight value of each candidate edge, and an action strategy is selected according to the comparison result through prior knowledge or neural network; finally, the parameters of the reinforcement learning model are determined through training and testing. We used four datasets to compare our method to the baseline method and conducted ablation experiments. On dataset NELL-995 and FB15k-237, the experimental results show that the MAP scores of our method are 87.8 and 45.2, and the optimal performance is achieved.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Bayesian Modeling and Causal Inference · Neural Networks and Applications · Reinforcement Learning in Robotics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Communication, National University of Defense Technology, Wuhan 430019, China
References (46)
Show source-provided references
1. Xiong, C., Merity, S., and Schoer, R. (2016, January 23–26). Dynamic memory networks for visual and textual question answering. Proceedings of the 33rd International Conference on Machine Learning, Honolulu, HI, USA.
2. Logan, R., Liu, N.F., and Peters, M.E. (2019, January 22–24). Using knowledge graphs for fact-aware language modeling. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.10.18653/v1/P19-1598 ↗
3. Xiong, W., Yu, S., and Guo, X. (2019, January 22–24). Improving question answering over incomplete KBs with knowledge-aware reader. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.10.18653/v1/P19-1417 ↗
4. Explainable AI: A review of machine learning interpretability methods10.3390/e23010018 ↗
5. Translating embeddings for modeling multi-relational data
6. Wang, Z., Zhang, J., and Feng, J. (2014, January 3–6). Knowledge graph embedding by translating on hyperplanes. Proceedings of the 28th Association for the Advancement of Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v28i1.8870 ↗
7. Lin, X., Liang, Y., Giunchiglia, F., Feng, X., and Guan, R. (2015). Compositional learning of relation path embedding for knowledge base completion. arXiv.
8. Ji, G., He, S., and Xu, L. (2015, January 26–31). Knowledge graph embedding via dynamic mapping matrix. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China.10.3115/v1/P15-1067 ↗
9. Pasquale, M., Pontus, S., and Sebastian, R. (2018). Convolutional 2D Knowledge Graph Embeddings. arXiv.
10. Trouillon, T., Welbl, J., and Riedel, S. (2016). Complex Embeddings for Simple Link Prediction. arXiv.
11. Lao, N., Mitchell, T., and Cohen, W.W. (2011, January 8–11). Rotate: Random walk inference and learning in a large scale knowledge base. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Scotland, UK.
12. Xiong, W., Hoang, T., and Wang, W.Y. (2017). Deeppath: A reinforcement learning method for knowledge graph reasoning. arXiv.10.18653/v1/D17-1060 ↗
13. Das, R., Dhuliawala, S., and Zaheer, M. (2018, January 28–30). Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. Proceedings of the ICLR, Vancouver, BC, Canada.
14. Wan, G., Pan, S., and Gong, C. (2021, January 7–15). Reasoning like human: Hierarchical reinforcement learning for knowledge graph reasoning. Proceedings of the 29th International Joint Conference on Artificial Intelligence, Yokohama, Japan.10.24963/ijcai.2020/267 ↗
15. Wang, H., Li, S., and Pan, R. (2019, January 3–7). Incorporating graph attention mechanism into knowledge graph reasoning based on deep reinforcement learning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1264 ↗
16. A framework for causal discovery in non-intervenable systems
17. Eric, J., Isabel, F., and Ilya, S. (2019). Auto-G-Computation of Causal Effects on a Network. arXiv.
18. Nicola, G., Jonas, P., and Sebastian, E. (2020). Causal discovery in heavy-tailed models. arXiv.
19. Large-scale chemical process causal discovery from big data with transformer-based deep learning10.1016/j.psep.2023.03.017 ↗
20. Semiparametric Proximal Causal Inference
21. Incorporating prior knowledge from counterfactuals into knowledge graph reasoning10.1016/j.knosys.2021.107035 ↗
22. Pearl, J., Creager, E., and Garg, A. (2016). Causal Inference in Statistics: A Primer, John Wiley & Sons. [1st ed.].
23. Pitis, S., Pan, S., and Gong, C. (2020, January 6–12). Counterfactual data augmentation using locally factored dynamics. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
24. Madumal, P., Miller, T., and Sonenberg, L. (2020, January 7–12). Explainable reinforcement learning through a causal lens. Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i03.5631 ↗
25. Lu, C.C., Huang, B.W., and Schölkopf, B. (2020). Sample-efficient reinforcement learning via counterfactual-based data augmentation. arXiv.
26. Buesing, L., Weber, T., and Zwols, Y. (2019, January 1–5). Woulda, coulda, shoulda: Counterfactually-guided policy search. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.
27. Model-based reinforcement learning: A survey10.1561/2200000086 ↗
28. Yi, F., Fu, W., and Liang, H. (2018, January 2–6). Model-based reinforcement learning: A survey. Proceedings of the 18th International Conference on Electronic Business, Guilin, China.
29. Convergence results for single-step on-policy reinforcement-learning algorithms10.1023/A:1007678930559 ↗
30. Q-learning10.1007/BF00992698 ↗
31. Mnih, V., Silver, D., and Graves, A. (2013). Playing Atari with deep reinforcement learning. arXiv.
32. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
33. Fortunato, M., Azar, M., and Piot, B. (2018, January 12-16). Noisy networks for exploration. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada.
34. Reinforcement learning: A survey10.1613/jair.301 ↗
35. Deep reinforcement learning: A survey10.1631/FITEE.1900533 ↗
36. Simple statistical gradient-following algorithms for connectionist reinforcement learning10.1007/BF00992696 ↗
37. Deng, Z., Jiang, J., Long, G., and Zhang, C. (2023). Causal Reinforcement Learning: A Survey. arXiv.
38. Maximilian, S., Bernhard, S., and Georg, M. (2016). Causal Reinforcement Learning: A Survey. arXiv.
39. Guo, J., Gong, M., and Tao, D. (2022, January 25–29). A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning. Proceedings of the 10th International Conference on Learning Representations (Virtual), Toulon, France.
40. Causal Confusion in Imitation Learning
41. False Correlation Reduction for Offline Reinforcement Learning10.1109/TPAMI.2023.3328397 ↗
42. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
43. Carlson, A., Betteridge, B., and Kisiel, B. (2010, January 10–15). Toward an architecture for never-ending language learning. Proceedings of the 25th AAAI Conference on Artificial Intelligence, Atlanta, GA, USA.
44. Bollacker, K., Evans, P., and Paritosh, T. (2008, January 6–10). Freebase: A collaboratively created graph database for structuring human knowledge. Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, New York, NY, USA.10.1145/1376616.1376746 ↗
45. Bordes, A., Weston, R., and Collobert, Y. (2011, January 16–22). Learning structured embeddings of knowledge bases. Proceedings of the 25th AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v25i1.7917 ↗
46. Sun, Z., Deng, Z.H., and Nie, J.Y. (2019, January 9–12). Rotate: Knowledge graph embedding by relational rotation in complex space. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.