Adaptive Position Control of Pneumatic Continuum Manipulator Based on MAML Meta-Reinforcement Learning
Source abstract
Abstract
Reinforcement learning algorithms usually focus on a specific task, which often performs well only in the training environment. When the task changes, its performance drops significantly, with the algorithm lacking the ability to adapt to new environments and tasks. For the position control of a pneumatic continuum manipulator (PCM), there is a high degree of similarity between tasks, and the training speed of new tasks can be accelerated by utilizing the training experience from other tasks. To increase the adaptability of control policies to new tasks, this paper proposes an adaptive position control algorithm of the PCM based on the Model-Agnostic Meta-Learning (MAML) meta-reinforcement learning algorithm. The MAML meta-reinforcement learning algorithm is used to train the control strategy of PCM, and the information and experience collected during the training process across multiple tasks are used to quickly learn the control policy for new tasks, improving the ability of PCM to quickly adapt to new tasks. The experimental results demonstrate that after training with the MAML meta-reinforcement learning algorithm, the PCM can significantly reduce the training time when faced with new tasks and obtain the control policies suitable for these new tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Oil and Gas Production Techniques · Intravenous Infusion Technology and Safety · Hydraulic and Pneumatic Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Mechanical Engineering and Automation, Northeastern University, Shenyang 110819, China
Funding
National Natural Science Foundation of China · National Natural Science Foundation of China for Youths
References (26)
Show source-provided references
1. You, X., Zhang, Y., Chen, X., Liu, X., Wang, Z., Jiang, H., and Chen, X. (2017, January 24–28). Model-free control for soft manipulators based on reinforcement learning. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8206123 ↗
2. Satheeshbabu, S., Uppalapati, X.K., Chowdhary, G., and Krishnan, G. (2019, January 20–24). Open loop position control of soft continuum arm using deep reinforcement learning. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793653 ↗
3. Satheeshbabu, S., Uppalapati, X.K., Fu, T., and Krishnan, G. (July, January 15). Continuous control of a soft continuum arm using deep reinforcement learning. Proceedings of the 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), New Haven, CO, USA.10.1109/RoboSoft48309.2020.9116003 ↗
4. Li, Y., Wang, X., and Kwok, K. (2022, January 23–27). Towards adaptive continuous control of soft robotic manipulator using reinforcement learning. Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan.10.1109/IROS47612.2022.9981335 ↗
5. Closed-loop dynamic control of a soft manipulator using deep reinforcement learning10.1109/LRA.2022.3146903 ↗
6. Pertsch, K., Lee, Y., and Lim, J.J. (2020). Accelerating reinforcement learning with learned skill priors. arXiv.
7. Rusu, A.A., Colmenarejo, C.G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., and Hadsell, R. (2015). Policy distillation. arXiv.
8. A survey on multi-task learning10.1109/TKDE.2021.3070203 ↗
9. Meta-learning in reinforcement learning
10. Duan, Y., Schulman, J., Chen, X., Bartlett, P.L., Sutskever, I., and Abbeel, P. (2016). RL2: Fast reinforcement learning via slow reinforcement learning. arXiv.
11. Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. (2017). A simple neural attentive meta-learner. arXiv.
12. Parisotto, E., Ghosh, S., Yalamanchi, S.B., Chinnaobireddy, V., Wu, Y., and Salakhutdinov, R. (2019). Concurrent meta reinforcement learning. arXiv.
13. Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. (2019, January 9–15). Efficient off-policy meta-reinforcement learning via probabilistic context variables. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.
14. Zhang, J., Wang, J., Hu, H., Chen, T., Chen, Y., Fan, C., and Zhang, C. (2021, January 18–24). MetaCURE: Meta reinforcement learning with empowerment-driven exploration. Proceedings of the 38th International Conference on Machine Learning, Virtual.
15. Fakoor, R., Chaudhari, P., Soatto, S., and Smola, A.J. (2019). Meta-Q-learning. arXiv.
16. Hu, S. (2023). Research of Meta-Reinforcement Learning and Its Application in Multi-Legged Robots. [Master’s Thesis, China University of Mining and Technology].
17. Finn, C., Abbeel, P., and Levine, S. (2017, January 6–11). Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International conference on machine learning, Sydney, Australia.
18. Song, X., Gao, W., Yang, Y., Choromanski, K., Pacchiano, A., and Tang, Y. (2019). ES-MAML: Simple hessian-free meta learning. arXiv.
19. Nichol, A., and Schulman, J. (2018). Reptile: A scalable metalearning algorithm. arXiv.
20. Gupta, A., Mendonca, R., Liu, Y.X., Abbeel, P., and Levine, S. (2018, January 3–8). Meta-reinforcement learning of structured exploration strategies. Proceedings of the 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, QC, Canada.
21. Liu, H., Socher, R., and Xiong, C. (2019, January 9–15). Taming MAML: Efficient unbiased meta-reinforcement learning. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.
22. A reaching skill learning method of manipulators based on meta-Q-learning and DDPG
23. Hao, T. (2022). Research on Meta-Optimizer and Meta-Reinforcement Learning Based on Bi-Level Optimization Meta-Learning. [Master’s Thesis, Shandong University].
24. Schoettler, G., Nair, A., Ojea, J.A., Levine, S., and Solowjow, E. (2020–24, January 24). Meta-reinforcement learning for robotic industrial insertion tasks. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9340848 ↗
25. Peng, K. (2022). Robot Motion Control Method Based on Particle Swarm Optimization and Meta-Reinforcement Learning. [Master’s Thesis, Yangzhou University].
26. Supervised meta-reinforcement learning with trajectory optimization for manipulation tasks10.1109/TCDS.2023.3286465 ↗