Safe Wasserstein Constrained Deep Q-Learning

2020-02-07 · Aaron Kandel, Scott J. Moura

This paper presents a distributionally robust Q-Learning algorithm (DrQ) which leverages Wasserstein ambiguity sets to provide idealistic probabilistic out-of-sample safety guarantees during online learning. First, we follow past work by separating the constraint functions from the principal objective to create a hierarchy of machines which estimate the feasible state-action space within the constrained Markov decision process (CMDP). DrQ works within this framework by augmenting constraint costs with tightening offset variables obtained through Wasserstein distributionally robust optimization (DRO). These offset variables correspond to worst-case distributions of modeling error characterized by the TD-errors of the constraint Q-functions. This procedure allows us to safely approach the nominal constraint boundaries. Using a case study of lithium-ion battery fast charging, we explore how idealistic safety guarantees translate to generally improved safety relative to conventional methods.

📄 PDF Abstract BibTeX arXiv:2002.03016

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learning

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Wasserstein Distributionally Robust Control Barrier Function using Conditional Value-at-Risk with Differentiable Convex Programming

2023-09-15 · Alaa Eddine Chriat, Chuangchuang Sun

Control Barrier functions (CBFs) have attracted extensive attention for designing safe controllers for their deployment in real-world safety-critical systems. However, the perception of the surrounding environment is oft…

Tail-Risk-Safe Monte Carlo Tree Search under PAC-Level Guarantees

2025-08-07 · Zuyuan Zhang, Arnob Ghosh, Tian Lan arxiv

Making decisions with respect to just the expected returns in Monte Carlo Tree Search (MCTS) cannot account for the potential range of high-risk, adverse outcomes associated with a decision. To this end, safety-aware MCT…

Fast Estimation of Wasserstein Distances via Regression on Sliced Wasserstein Distances

2025-09-24 · Khai Nguyen, Hai Nguyen, Nhat Ho arxiv

We address the problem of efficiently computing Wasserstein distances for multiple pairs of distributions drawn from a meta-distribution. To this end, we propose a fast estimation method based on regressing Wasserstein d…

Point Clouds

Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraints

2024-11-28 · Pekka Malo, Lauri Viitasaari, Antti Suominen, Eeva Vilkkumaa 외

This paper studies reinforcement learning (RL) in infinite-horizon dynamic decision processes with almost-sure safety constraints. Such safety-constrained decision processes are central to applications in autonomous syst…

Decision Making Reinforcement Learning (RL)

Certified Safety Radii in Forecast-Error Space for Wasserstein Distributionally Robust Small Signal Stability-Constrained AC Optimal Power Flow via Lifted Spectrahedral Containment

2026-08-31 · Ziqi Zhang, Xi Chen arxiv

Directly robustifying small-signal stability in AC optimal power flow is challenging since the stability boundary in the original uncertainty space is implicit, highly nonconvex, and changes with the operating decision. …