A Deep Reinforcement Learning Design for Virtual Synchronous Generators Accommodating Modular Multilevel Converters
Source abstract
Abstract
The deep reinforcement learning (DRL) technique has gained attention for its potential in designing “virtual network” controllers. This skill utilizes a novel solution that can avoid the specific parameters and system model required in classical dynamic programming algorithms. However, addressing the issue of system uncertainties and performance deterioration remains a challenge. To overcome this challenge, the authors propose a new control prototype using a twin delayed deep deterministic policy gradient (TD3)-based adaptive controller, which replaces the conventional virtual synchronous generator (VSG) module in the modular multilevel converter (MMC) control. In this approach, an adaptive programming module is developed using a critic fuzzy network point of view to determine the optimal control policy. The modification presented in this framework is able to improve the system stability and resist disruptions while retaining the merits of the conventional VSG control model. The proposed approach is implemented and tested using the DRL toolbox in MATLAB/Simulink.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Microgrid Control and Optimization · HVDC Systems and Fault Protection · Power System Optimization and Stability
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electrical Engineering, China University of Mining and Technology, Xuzhou 221000, China
Funding
National Key Research and Development Program of China
References (26)
Show source-provided references
1. Lv, Z., and Zhong, Q.C. (2017, January 16–20). Control of modular multilevel converters as virtual synchronous machines. Proceedings of the IEEE Power & Energy Society General Meeting (PESGM), Chicago, IL, USA.10.1109/PESGM.2017.8274551 ↗
2. A parameter alternating VSG controller of VSC-MTDC systems for low frequency oscillation damping10.1109/TPWRS.2020.2997859 ↗
3. Synchronverters: Inverters that mimic synchronous generators10.1109/TIE.2010.2048839 ↗
4. Virtual inertial frequency control strategy for micro grid power supply
5. Impedance modeling and grid connection stability analysis of grid voltage feedforward control VSG
6. Model-free volt-var control based on measurement data analytics10.1109/TPWRS.2018.2874543 ↗
7. Rehman, H.U. (2021). Study on VSG Stability Control and Primary Frequency Modulation of Renewable Energy Generation Unit. [Ph.D. Dissertation, North China Electric Power University].
8. VSG control strategy based on improved particle swarm optimization algorithm
9. Wanigasekara, C., Swain, A., Nguang, S.K., and Prusty, B.G. (2018, January 8–13). Improved Learning from Small Data Sets Through Effective Combination of Machine Learning Tools with VSG Techniques. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.10.1109/IJCNN.2018.8489759 ↗
10. Zheng, J., Wang, A., Zhang, T., Zhang, H., and Gu, N. (2019, January 27–30). Improved Adaptive Control Strategy of Inertia for Virtual Synchronous Generator Based on Fuzzy Control. Proceedings of the 2019 Chinese Control Conference (CCC), Guangzhou, China.10.23919/ChiCC.2019.8866600 ↗
11. Neuronal reward and decision signals: From theories to data10.1152/physrev.00023.2014 ↗
12. Deep reinforcement learning based approach for optimal power flow of distribution networks embedded with renewable energy and storage devices10.35833/MPCE.2020.000557 ↗
13. Review on the research and practice of deep learning and reinforcement learning in smart grids10.17775/CSEEJPES.2018.00520 ↗
14. Sutton Richard, S., and Barto Andrew, G. (2018). Reinforcement Learning: An Introduction, MIT Press. [2nd ed.].
15. A Survey on Deep Reinforcement Learning
16. Deep reinforcement learning: A brief survey10.1109/MSP.2017.2743240 ↗
17. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
18. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2021, November 07). Continuous Control with Deep Reinforcement Learning. Available online: http://arxiv.org/abs/1509.029711.
19. Wu, W., Guo, F., Ni, Q., Liu, X., Qiu, L., and Fang, Y. (2022, January 28–31). Deep Q-Network based Adaptive Robustness Parameters for Virtual Synchronous Generator. Proceedings of the 2022 IEEE Transportation Electrification Conference and Expo, Asia-Pacific (ITEC Asia-Pacific), Haining, China.10.1109/ITECAsia-Pacific56316.2022.9941893 ↗
20. Deep reinforcement learning based parameter self-tuning control strategy for VSG10.1016/j.egyr.2022.02.147 ↗
21. Optimization of MMC Voltage Equalization Control Parameters
22. Li, J. (2019). Design and Application of Modern Synchronous Generator Excitation Systems, IEEE.10.1002/9781118841006 ↗
23. Common-mode voltage injection-based nearest level modulation with loss reduction for modular multilevel converters10.1049/iet-rpg.2015.0438 ↗
24. Zheng, X. (2013). Flexible High Voltage Direct Current System, China Machine Press.
25. Fujimoto, S., Hoof, H., and Meger, D. (2018, January 10–15). Addressing Function Approximation Error in Actor-Critic Methods. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
26. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21–26). Deterministic policy gradient algorithms. Proceedings of the International Conference on Machine Learning, Beijing, China.