Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms
Classical reinforcement learning (RL) aims to optimize the expected cumulative reward. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative reward. We parameterize the policy controlling actions by neural networks, and propose a novel policy gradient algorithm called Quantile-Based Policy Optimization (QPO) and its variant Quantile-Based Proximal Policy Optimization (QPPO) for solving deep RL problems with quantile objectives. QPO uses two coupled iterations running at different timescales for simultaneously updating quantiles and policy parameters, whereas QPPO is an off-policy version of QPO that allows multiple updates of parameters during one simulation episode, leading to improved algorithm efficiency. Our numerical results indicate that the proposed algorithms outperform the existing baseline algorithms under the quantile criterion.
Code (1)
Tasks
Deep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Vocal Bursts Valence PredictionSimilar Papers 제목 키워드 기반
Stochastic Approximation Methods for Distortion Risk Measure Optimization
Distortion Risk Measures (DRMs) capture risk preferences in decision-making and serve as general criteria for managing uncertainty. This paper proposes gradient descent algorithms for DRM optimization based on two dual r…
Reinforcement LearningDensity EstimationQuantile-Based Policy Optimization for Reinforcement Learning
Classical reinforcement learning (RL) aims to optimize the expected cumulative rewards. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative rewards. We parameterize the p…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Gradient Temporal Difference with Momentum: Stability and Convergence
Gradient temporal difference (Gradient TD) algorithms are a popular class of stochastic approximation (SA) algorithms used for policy evaluation in reinforcement learning. Here, we consider Gradient TD algorithms with an…
Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning
In multi-timescale multi-agent reinforcement learning (MARL), agents interact across different timescales. In general, policies for time-dependent behaviors, such as those induced by multiple timescales, are non-stationa…
energy managementInductive BiasManagementMulti-agent Reinforcement Learning+1Sample Complexity Bounds for Two Timescale Value-based Reinforcement Learning Algorithms
Two timescale stochastic approximation (SA) has been widely used in value-based reinforcement learning algorithms. In the policy evaluation setting, it can model the linear and nonlinear temporal difference learning with…
reinforcement-learningReinforcement Learning (RL)Vocal Bursts Valence Prediction