Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
Source abstract
Abstract
In vertical vehicle dynamics control, semi-active dampers are used to enhance ride comfort and road-holding with only minor additional energy expenses. However, a complex control problem arises from the combined effects of (1) the constrained semi-active damper characteristic, (2) the opposing control objectives of improving ride comfort and road-holding, and (3) the additionally coupled vertical dynamic system. This work presents the application of Reinforcement Learning to the vertical dynamics control problem of a real street vehicle to address these issues. We discuss the entire Reinforcement Learning-based controller design process, which started with deriving a sufficiently accurate training model representing the vehicle behavior. The obtained model was then used to train a Reinforcement Learning agent, which offered improved vehicle ride qualities. After that, we verified the trained agent in a full-vehicle simulation setup before the agent was deployed in the real vehicle. Quantitative and qualitative real-world tests highlight the increased performance of the trained agent in comparison to a benchmark controller. Tests on a real-world four-post test rig showed that the trained RL-based controller was able to outperform an offline-optimized benchmark controller on road-like excitations, improving the comfort criterion by about 2.5% and the road-holding criterion by about 2.0% on average.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Vehicle Dynamics and Control Systems · Vibration Control and Rheological Fluids · Spinal Cord Injury Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of System Dynamics and Control, German Aerospace Center (DLR), 82234 Weßling, Germany
Funding
German Federal Ministry of Education and Research · DLR basic funding
References (45)
Show source-provided references
1. Vibration Control Using Semi-Active Force Generators10.1115/1.3438373 ↗
2. Extended Ground-Hook—New Concept of Semi-Active Control of Truck’s Suspension10.1080/00423119708969333 ↗
3. Survey and performance evaluation on some automotive semi-active suspension control methods: A comparative study on a single-corner model10.1016/j.arcontrol.2012.03.011 ↗
4. Mixed Sky-Hook and ADD: Approaching the Filtering Limits of a Semi-Active Suspension10.1115/1.2745846 ↗
5. DTC: Deep Tracking Control10.1126/scirobotics.adh5401 ↗
6. An Introduction to Deep Reinforcement Learning10.1561/2200000071 ↗
7. Ruggaber, J., Ahmic, K., Brembeck, J., Baumgartner, D., and Tobolář, J. (2023). AI-For-Mobility—A New Research Platform for AI-Based Control Methods. Appl. Sci., 13.10.3390/app13052879 ↗
8. Continuous action reinforcement learning applied to vehicle suspension control10.1016/S0957-4158(97)00003-2 ↗
9. Tognetti, S., Savaresi, S., Spelta, C., and Restelli, M. (2009, January 8–10). Batch Reinforcement Learning for semi-active suspension control. Proceedings of the 2009 IEEE International Conference on Control Applications (CCA), St. Petersburg, Russia.10.1109/CCA.2009.5281070 ↗
10. Pfeffer, P. (2017). Explicit model predictive control of semi-active suspension systems using Artificial Neural Networks (ANN). 8th International Munich Chassis Symposium, Springer Fachmedien Wiesbaden.
11. Enhancing skyhook for semi-active suspension control via machine learning10.1016/j.ifacsc.2021.100161 ↗
12. Semi-Active Suspension Control Based on Deep Reinforcement Learning10.1109/ACCESS.2020.2964116 ↗
13. Liang, G., Zhao, T., and Wei, Y. (2021, January 29–31). DDPG based self-learning active and model-constrained semi-active suspension control. Proceedings of the IEEE 5th CAA International Conference on Vehicular Control and Intelligence (CVCI), Tianjin, China.10.1109/CVCI54083.2021.9661158 ↗
14. Han, S.-Y., and Liang, T. (2022). Reinforcement-Learning-Based Vibration Control for a Vehicle Semi-Active Suspension System via the PPO Approach. Appl. Sci., 12.10.3390/app12063078 ↗
15. Kim, S., Kim, C., Shin, S., and Kim, S.-W. (2023, January 5–8). Deep Reinforcement Learning for Semi-Active Suspension: A Feasibility Study. Proceedings of the IEEE 2023 International Conference on Electronics, Information, and Communication (ICEIC), Singapore.10.1109/ICEIC57457.2023.10049850 ↗
16. Wang, Y., Wang, C., Zhao, S., and Guo, K. (2023). Research on Deep Reinforcement Learning Control Algorithm for Active Suspension Considering Uncertain Time Delay. Sensors, 23.10.3390/s23187827 ↗
17. Deep Reinforcement Learning of Semi-Active Suspension Controller for Vehicle Ride Comfort10.1109/TVT.2022.3207510 ↗
18. Suspension Control Strategies Using Switched Soft Actor-Critic Models for Real Roads10.1109/TIE.2022.3153805 ↗
19. (2016). Mechanical Vibration—Road Surface Profiles—Reporting of Measured Data. Standard No. ISO 8608:2016.
20. Ultsch, J., Tobolar, J., Ruggaber, J., Pfeiffer, A., Kamp, T., Brembeck, J., Baumgartner, D., Ziegler, M., Wahl, U., and Rohn, M. (2023). Sachbericht zum Projekt “KI-basierte Fahrwerksregelung KIFAHR”, DLR Institut für Systemdynamik und Regelungstechnik.
21. Peng, X., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018, January 21–25). Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. Proceedings of the 2018 IEEE International Conference on Robotics and Automation, ICRA, Brisbane, Australia.10.1109/ICRA.2018.8460528 ↗
22. Modelica Association (2024, June 05). Modelica—A Unified Object-Oriented Language for Systems Modeling—Language Specification. Available online: https://specification.modelica.org/maint/3.6/MLS.html.
23. Modelica Association (2024, June 12). Functional Mock-Up Interface Specification. Available online: https://fmi-standard.org/.
24. Ultsch, J., Ruggaber, J., Pfeiffer, A., Schreppel, C., Tobolář, J., Brembeck, J., and Baumgartner, D. (2021). Advanced Controller Development Based on eFMI with Applications to Automotive Vertical Dynamics Control. Actuators, 10.10.3390/act10110301 ↗
25. Fleps-Dezasse, M., Tobolar, J., and Pitzer, J. (2014, January 10–12). Modelling and parameter identification of a semi-active vehicle damper. Proceedings of the 10th International Modelica Conference, Lund, Sweden.10.3384/ecp14096283 ↗
26. Pfeiffer, A. (2012, January 3–5). Optimization Library for Interactive Multi-Criteria Optimization Tasks. Proceedings of the 9th International Modelica Conference, Munich, Germany.10.3384/ecp12076669 ↗
27. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.].
28. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv.
29. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv.
30. Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016, January 2–4). Continuous control with deep reinforcement learning. Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, PR, USA.
31. Dy, J., and Krause, A. (2018, January 10–15). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning, ICML, Stockholm, Sweden.
32. Dy, J., and Krause, A. (2018, January 10–15). RLlib: Abstractions for Distributed Reinforcement Learning. Proceedings of the 35th International Conference on Machine Learning, ICML, Stockholm, Sweden.
33. Stable-Baselines3: Reliable Reinforcement Learning Implementations
34. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv.
35. Towers, M., Terry, J., Kwiatkowski, A., Balis, J., Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Arjun, K., and Krimmel, M. (2024, June 05). Gymnasium. Available online: https://zenodo.org/records/8127026.
36. Ultsch, J., Brembeck, J., and de Castro, R. (2019). Learning-Based Path Following Control for an Over-Actuated Robotic Vehicle. VDI-AUTOREG, VDI Verlag.10.51202/9783181023495-25 ↗
37. Ultsch, J., Mirwald, J., Brembeck, J., and de Castro, R. (November, January 19). Reinforcement Learning-based Path Following Control for a Vehicle with Variable Delay in the Drivetrain. Proceedings of the 2020 IEEE Intelligent Vehicles Symposium, IV, Las Vegas, NV, USA.10.1109/IV47402.2020.9304578 ↗
38. Raffin, A. (2024, June 12). RL Baselines3 Zoo. Available online: https://github.com/DLR-RM/rl-baselines3-zoo.
39. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., and Abbeel, P. (2019). Soft Actor-Critic Algorithms and Applications. arXiv.
40. Overcoming catastrophic forgetting in neural networks10.1073/pnas.1611835114 ↗
41. Savaresi, S., Poussot-Vassal, C., Spelta, C., Sename, O., and Dugard, L. (2010). Semi-Active Suspension Control Design for Vehicles, Butterworth-Heinemann.10.1016/B978-0-08-096678-6.00002-X ↗
42. Objectification methods for ride comfort10.1007/s10010-019-00361-6 ↗
43. (1997). Mechanical Vibration and Shock—Evaluation of Human Exposure to Whole-Body Vibration. Part 1: General Requirements. Standard No. ISO 2631-1:1997.
44. Orlova, A., and Cole, D. (2022). Modelling and Validation of the TMeasy Tyre Model for Extreme Parking Manoeuvres. Advances in Dynamics of Vehicles on Roads and Tracks II, Springer International Publishing.
45. Simulated Road Profiles According to ISO 8608 in Vibration Analysis