Augmented Lagrangian-Based Safe Reinforcement Learning Approach for Distribution System Volt/VAR Control
This paper proposes a data-driven solution for Volt-VAR control problem in active distribution system. As distribution system models are always inaccurate and incomplete, it is quite difficult to solve the problem. To handle with this dilemma, this paper formulates the Volt-VAR control problem as a constrained Markov decision process (CMDP). By synergistically combining the augmented Lagrangian method and soft actor critic algorithm, a novel safe off-policy reinforcement learning (RL) approach is proposed in this paper to solve the CMDP. The actor network is updated in a policy gradient manner with the Lagrangian value function. A double-critics network is adopted to synchronously estimate the action-value function to avoid overestimation bias. The proposed algorithm does not require strong convexity guarantee of examined problems and is sample efficient. A two-stage strategy is adopted for offline training and online execution, so the accurate distribution system model is no longer needed. To achieve scalability, a centralized training distributed execution strategy is adopted for a multi-agent framework, which enables a decentralized Volt-VAR control for large-scale distribution system. Comprehensive numerical experiments with real-world electricity data demonstrate that our proposed algorithm can achieve high solution optimality and constraints compliance.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement Learning (RL)Safe Reinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?
Diffusion policy sampling enables reinforcement learning (RL) to represent multimodal action distributions beyond suboptimal unimodal Gaussian policies. However, existing diffusion-based RL methods primarily focus on off…
Reinforcement LearningAugmented Lagrangian Multiplier Network for State-wise Safety in Reinforcement Learning
Safety is a primary challenge in real-world reinforcement learning (RL). Formulating safety requirements as state-wise constraints has become a prominent paradigm. Handling state-wise constraints with the Lagrangian meth…
Reinforcement LearningCancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes
Constrained Markov Decision Processes (CMDPs) are one of the common ways to model safe reinforcement learning problems, where constraint functions model the safety objectives. Lagrangian-based dual or primal-dual algorit…
Safe Reinforcement LearningEnhance the Safety in Reinforcement Learning by ADRC Lagrangian Methods
Safe reinforcement learning (Safe RL) seeks to maximize rewards while satisfying safety constraints, typically addressed through Lagrangian-based methods. However, existing approaches, including PID and classical Lagrang…
Reinforcement LearningLagrangian-based online safe reinforcement learning for state-constrained systems
This paper proposes a safe reinforcement learning (RL) algorithm that approximately solves the state-constrained optimal control problem for continuous-time uncertain nonlinear systems. We formulate the safe RL problem a…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning