A Parametric Study of a Deep Reinforcement Learning Control System Applied to the Swing-Up Problem of the Cart-Pole
Source abstract
Abstract
In this investigation, the nonlinear swing-up problem associated with the cart-pole system modeled as a multibody dynamical system is solved by developing a deep Reinforcement Learning (RL) controller. Furthermore, the sensitivity analysis of the deep RL controller applied to the cart-pole swing-up problem is carried out. To this end, the influence of modifying the physical properties of the system and the presence of dry friction forces are analyzed employing the cumulative reward during the task. Extreme limits for the modifications of the parameters are determined to prove that the neural network architecture employed in this work features enough learning capability to handle the task under modifications as high as 90% on the pendulum mass, as well as a 100% increment on the cart mass. As expected, the presence of dry friction greatly affects the performance of the controller. However, a post-training of the agent in the modified environment takes only thirty-nine episodes to find the optimal control policy, resulting in a promising path for further developments of robust controllers.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Real-time simulation and control systems · Vehicle Dynamics and Control Systems · Traffic control and management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
MEID4 Academic Spin-Off of the University of Salerno, via Giovanni Paolo II, 132, 84084 Fisciano (SA), Italy
Department of Industrial Engineering, University of Salerno, via Giovanni Paolo II, 132, 84084 Fisciano (SA), Italy
References (75)
Show source-provided references
1. A Reinforcement Learning Strategy for the Swing-Up of the Double Pendulum on a Cart10.1016/j.promfg.2018.06.004 ↗
2. Manrique, C., Pappalardo, C.M., and Guida, D. (2019). A Model Validating Technique for the Kinematic Study of Two-Wheeled Vehicles. Lect. Notes Mech. Eng., 549–558.10.1007/978-3-030-40724-7_56 ↗
3. Multibody modeling and nonlinear control of the pantograph/catenary system10.1007/s00419-019-01530-3 ↗
4. Pappalardo, C., and Guida, D. (2019). Forward and Inverse Dynamics of a Unicycle-Like Mobile Robot. Machines, 7.10.3390/machines7010005 ↗
5. Villecco, F., and Pellegrino, A. (2017). Evaluation of Uncertainties in the Design Process of Complex Mechanical Systems. Entropy, 19.10.3390/e19090475 ↗
6. Villecco, F., and Pellegrino, A. (2017). Entropic Measure of Epistemic Uncertainties in Multibody System Models by Axiomatic Design. Entropy, 19.10.3390/e19070291 ↗
7. Hu, D., Pei, Z., and Tang, Z. (2020). Single-Parameter-Tuned Attitude Control for Quadrotor with Unknown Disturbance. Appl. Sci., 10.10.3390/app10165564 ↗
8. Talamini, J., Bartoli, A., De Lorenzo, A.D., and Medvet, E. (2020). On the Impact of the Rules on Autonomous Drive Learning. Appl. Sci., 10.10.3390/app10072394 ↗
9. Sharifzadeh, S., Chiotellis, I., Triebel, R., and Cremers, D. (2016). Learning to drive using inverse reinforcement learning and deep q-networks. arXiv.
10. Cho, N.J., Lee, S.H., Kim, J.B., and Suh, I.H. (2020). Learning, Improving, and Generalizing Motor Skills for the Peg-in-Hole Tasks Based on Imitation Learning and Self-Learning. Appl. Sci., 10.10.3390/app10082719 ↗
11. Zhang, H., Qu, C., Zhang, J., and Li, J. (2020). Self-Adaptive Priority Correction for Prioritized Experience Replay. Appl. Sci., 10.10.3390/app10196925 ↗
12. Hong, D., Kim, M., and Park, S. (2020). Study on Reinforcement Learning-Based Missile Guidance Law. Appl. Sci., 10.10.3390/app10186567 ↗
13. Rivera, Z.B., De Simone, M.C., and Guida, D. (2019). Unmanned Ground Vehicle Modelling in Gazebo/ROS-Based Environments. Machines, 7.10.3390/machines7020042 ↗
14. Control design for an under-actuated UAV model10.5937/fmet1804443D ↗
15. A model for system uncertainty in reinforcement learning10.1016/j.sysconle.2018.09.011 ↗
16. Reinforcement learning is direct adaptive optimal control10.1109/37.126844 ↗
17. Cheng, Q., Wang, X., Niu, Y., and Shen, L. (2018). Reusing Source Task Knowledge via Transfer Approximator in Reinforcement Transfer Learning. Symmetry, 11.10.3390/sym11010025 ↗
18. Nichols, B.D. (2015, January 9–12). Continuous Action-Space Reinforcement Learning Methods Applied to the Minimum-Time Swing-Up of the Acrobot. Proceedings of the 2015 IEEE International Conference on Systems, Man, and Cybernetics, Hong Kong, China.10.1109/SMC.2015.364 ↗
19. State representation learning for control: An overview10.1016/j.neunet.2018.07.006 ↗
20. Oh, E. (2020). Reinforcement-Learning-Based Virtual Energy Storage System Operation Strategy for Wind Power Forecast Uncertainty Management. Appl. Sci., 10.10.3390/app10186420 ↗
21. Phan, B.C., and Lai, Y.-C. (2019). Control Strategy of a Hybrid Renewable Energy System Based on Reinforcement Learning Approach for an Isolated Microgrid. Appl. Sci., 9.10.3390/app9194001 ↗
22. Kővári, B., Hegedüs, F., and Bécsi, T. (2020). Design of a Reinforcement Learning-Based Lane Keeping Planning Agent for Automated Vehicles. Appl. Sci., 10.10.3390/app10207171 ↗
23. Tran, D.Q., and Bae, S.-H. (2020). Proximal Policy Optimization Through a Deep Reinforcement Learning Framework for Multiple Autonomous Vehicles at a Non-Signalized Intersection. Appl. Sci., 10.10.3390/app10165722 ↗
24. Sattarov, O., Muminov, A., Lee, C.W., Kang, H.K., Oh, R., Ahn, J., Oh, H.J., and Jeon, H.S. (2020). Recommending Cryptocurrency Trading Points with Deep Reinforcement Learning Approach. Appl. Sci., 10.10.3390/app10041506 ↗
25. Rundo, F. (2019). Deep LSTM with Reinforcement Learning Layer for Financial Trend Prediction in FX High Frequency Trading Systems. Appl. Sci., 9.10.3390/app9204460 ↗
26. Chen, H., Liu, Y., Zhou, Z., and Zhang, M. (2020). A2C: Attention-Augmented Contrastive Learning for State Representation Extraction. Appl. Sci., 10.10.3390/app10175902 ↗
27. Xiang, G., and Su, J. (2019). Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control. IEEE Trans. Cybern., 1–14.
28. Control Policy with Autocorrelated Noise in Reinforcement Learning for Robotics10.7763/IJMLC.2015.V5.489 ↗
29. Beltran-Hernandez, C.C., Petit, D., Ramirez-Alpizar, I.G., and Harada, K. (2020). Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep-Reinforcement-Learning Approach. Appl. Sci., 10.10.3390/app10196923 ↗
30. Moreira, I., Rivas, J., Cruz, F., Dazeley, R., Ayala, A., and Fernandes, B. (2020). Deep Reinforcement Learning with Interactive Feedback in a Human–Robot Environment. Appl. Sci., 10.10.3390/app10165574 ↗
31. Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J.M., Boots, B., and Theodorou, E.A. (June, January 29). Information theoretic MPC for model-based reinforcement learning. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989202 ↗
32. Anderson, C.W., Lee, M., and Elliott, D.L. (2015, January 12–16). Faster reinforcement learning after pretraining deep networks to predict state dynamics. Proceedings of the 2015 International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland.10.1109/IJCNN.2015.7280824 ↗
33. Lee, M., and Anderson, C.W. (2014, January 9–12). Convergent reinforcement learning control with neural networks and continuous action search. Proceedings of the 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), Orlando, FL, USA.10.1109/ADPRL.2014.7010612 ↗
34. Maei, H.R., Szepesvari, C., Bhatnagar, S., Precup, D., Silver, D., and Sutton, R.S. (2009, January 7–10). Convergent temporal-difference learning with arbitrary smooth function approximation. Proceedings of the 22nd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
35. Robust Reinforcement Learning10.1162/0899766053011528 ↗
36. An Introduction to Deep Reinforcement Learning10.1561/2200000071 ↗
37. Yang, Y., Li, X., and Zhang, L. (2016, January 12–15). Task-specific pre-learning to improve the convergence of reinforcement learning based on a deep neural network. Proceedings of the 2016 12th World Congress on Intelligent Control and Automation (WCICA), Guilin, China.10.1109/WCICA.2016.7578787 ↗
38. Back to reality: Crossing the reality gap in evolutionary robotics10.1016/S1474-6670(17)32084-0 ↗
39. Bekar, C., Yuksek, B., and Inalhan, G. (2020, January 6–10). High Fidelity Progressive Reinforcement Learning for Agile Maneuvering UAVs. Proceedings of the AIAA Scitech 2020 Forum; American Institute of Aeronautics and Astronautics, Orlando, FL, USA.10.2514/6.2020-0898 ↗
40. Boubaker, O., and Iriarte, R. (2017). The Inverted Pendulum in Control Theory and Robotics: From Theory to New Innovations, IET.10.1049/PBCE111E ↗
41. Fast hybrid dual mode NMPC for a parallel double inverted pendulum with experimental validation10.1049/iet-cta.2020.0130 ↗
42. Digital implementation of a continuous-time nonlinear optimal controller: An experimental study with real-time computations10.1016/j.isatra.2020.01.020 ↗
43. Robust Control for Switched Systems with Unmatched Uncertainties Based on Switched Robust Integral Sliding Mode10.1109/ACCESS.2020.3012462 ↗
44. Robust Backstepping Sliding Mode Control Design for a Class of Underactuated Electro–Mechanical Nonlinear Systems10.1007/s42835-020-00436-3 ↗
45. Robust stabilization of underactuated nonlinear systems: A fast terminal sliding mode approach10.1016/j.isatra.2016.10.017 ↗
46. An adaptive swing-up sliding mode controller design for a real inverted pendulum system based on Culture-Bees algorithm10.1016/j.ejcon.2018.12.001 ↗
47. Design of hybrid sliding mode controller based on fireworks algorithm for nonlinear inverted pendulum systems10.1177/1687814016684273 ↗
48. Mansoor, H., and Bhutta, H.A. (2016, January 11–12). Genetic algorithm based optimal back stepping controller design for stabilizing inverted pendulum. Proceedings of the 2016 International Conference on Computing, Electronic and Electrical Engineering (ICE Cube), Quetta, Pakistan.10.1109/ICECUBE.2016.7495252 ↗
49. Real-Time Swing-up and Stabilization Control of a Cart-Pendulum System with Constrained Cart Movement10.1515/ijnsns-2017-0040 ↗
50. A Self-learning Robust Swing-up algorithm10.1177/0142331215584420 ↗
51. Optimizing the double inverted pendulum’s performance via the uniform neuro multiobjective genetic algorithm10.1007/s11633-017-1069-8 ↗
52. Zheng, Y., Li, X., and Xu, L. (2020). Balance Control for the First-order Inverted Pendulum Based on the Advantage Actor-critic Algorithm. Int. J. Control. Autom. Syst.10.1007/s12555-019-0278-z ↗
53. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. arXiv.
54. Imitation Reinforcement Learning-Based Remote Rotary Inverted Pendulum Control in OpenFlow Network10.1109/ACCESS.2019.2905621 ↗
55. Dao, P.N., and Liu, Y.-C. (2020). Adaptive Reinforcement Learning Strategy with Sliding Mode Control for Unknown and Disturbed Wheeled Inverted Pendulum. Int. J. Control. Autom. Syst.10.1007/s12555-019-0912-9 ↗
56. Genetic Algorithm-Optimized Fuzzy Lyapunov Reinforcement Learning for Nonlinear Systems10.1007/s13369-019-04126-9 ↗
57. Benchmarking model-free and model-based optimal control10.1016/j.robot.2017.02.006 ↗
58. Efficient Model Learning Methods for Actor—Critic Control10.1109/TSMCB.2011.2170565 ↗
59. Maity, S., and Luecke, G.R. (2019). Stabilization and Optimization of Design Parameters for Control of Inverted Pendulum. J. Dyn. Syst. Meas. Control, 141.10.1115/1.4042953 ↗
60. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press. [2nd ed.].
61. Brachman, R.J., and Dietterich, T. (2010). Algorithms for Reinforcement Learning, Morgan & Claypool Publishers. [1st ed.].
62. Lillicrap, T.P., Hunt, J.J., Pritzel, A.P., Heess, N.H., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous Control with Deep Reinforcement Learning. arXiv.
63. The Arcade Learning Environment: An Evaluation Platform for General Agents10.1613/jair.3912 ↗
64. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
65. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv.
66. Boubaker, O. (2012, January 1–3). The inverted pendulum: A fundamental benchmark in control theory and robotics. Proceedings of the International Conference on Education and e-Learning Innovations, Sousse, Tunisia.10.1109/ICEELI.2012.6360606 ↗
67. Nagendra, S., Podila, N., Ugarakhod, R., and George, K. (2017, January 13–16). Comparison of reinforcement learning algorithms applied to the cart-pole problem. Proceedings of the 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Udupi, India.10.1109/ICACCI.2017.8125811 ↗
68. Wawrzynski, P. (2007, January 9–12). Learning to Control a 6-Degree-of-Freedom Walking Robot. Proceedings of the EUROCON 2007—The International Conference on “Computer as a Tool”, Warsaw, Poland.10.1109/EURCON.2007.4400335 ↗
69. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv.
70. Riedmiller, M., Peters, J., and Schaal, S. (2007, January 1–5). Evaluation of Policy Gradient Methods and Variants on the Cart-Pole Benchmark. Proceedings of the 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, Honolulu, HI, USA.10.1109/ADPRL.2007.368196 ↗
71. Multibody dynamics simulation of planar linkages with Dahl friction10.1007/s11044-007-9047-5 ↗
72. Review and comparison of dry friction force models10.1007/s11071-015-2485-3 ↗
73. Kikuuwe, R., Takesue, N., Sano, A., Mochiyama, H., and Fujimoto, H. (2005, January 2–6). Fixed-step friction simulation: From classical Coulomb model to modern continuous models. Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, AB, Canada.10.1109/IROS.2005.1545579 ↗
74. De Simone, M.C., and Guida, D. (2020). Experimental investigation on structural vibrations by a new shaking table. Lect. Notes Mech. Eng., 819–831.10.1007/978-3-030-41057-5_66 ↗
75. Stability analysis of rigid multibody mechanical systems with holonomic and nonholonomic constraints10.1007/s00419-020-01706-2 ↗