Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning
Safe reinforcement learning (RL) is a popular and versatile paradigm to learn reward-maximizing policies with safety guarantees. Previous works tend to express the safety constraints in an expectation form due to the ease of implementation, but this turns out to be ineffective in maintaining safety constraints with high probability. To this end, we move to the quantile-constrained RL that enables a higher level of safety without any expectation-form approximations. We directly estimate the quantile gradients through sampling and provide the theoretical proofs of convergence. Then a tilted update strategy for quantile gradients is implemented to compensate the asymmetric distributional density, with a direct benefit of return performance. Experiments demonstrate that the proposed model fully meets safety requirements (quantile constraints) while outperforming the state-of-the-art benchmarks with higher return.
Code (1)
Tasks
Formreinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningSimilar Papers 제목 키워드 기반
Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability
Constrained reinforcement learning (RL) is an area of RL whose objective is to find an optimal policy that maximizes expected cumulative return while satisfying a given constraint. Most of the previous constrained RL wor…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms
Classical reinforcement learning (RL) aims to optimize the expected cumulative reward. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative reward. We parameterize the pol…
Deep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Vocal Bursts Valence PredictionOn Learning the Tail Quantiles of Driving Behavior Distributions via Quantile Regression and Flows
Towards safe autonomous driving (AD), we consider the problem of learning models that accurately capture the diversity and tail quantiles of human driver behavior probability distributions, in interaction with an AD vehi…
Autonomous DrivingDiversityquantile regressionregressionGenerative Regression with IQ-BART
Implicit Quantile BART (IQ-BART) posits a non-parametric Bayesian model on the conditional quantile function, acting as a model over a conditional model for $Y$ given $X$. One of the key ingredients is augmenting the obs…
Time Series ForecastingGradient Equilibrium in Online Learning: Theory and Applications
We present a new perspective on online learning that we refer to as gradient equilibrium: a sequence of iterates achieves gradient equilibrium if the average of gradients of losses along the sequence converges to zero. I…
Learning Theory