Samples Are Useful? Not Always: denoising policy gradient updates using variance explained
Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based solely on sampling the agent's policy, other measures directly accessible through these algorithms could be used to improve sampling before each policy update. Following this line of thoughts, we propose the use of SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. This criterion, the fraction of variance explained Vex, is a measure of the discrepancy between a model and actual samples. In this work, Vex is used to evaluate the impact each transition will have on learning: this criterion refines sampling and improves the policy gradient algorithm. In this paper: (a) We introduce and explore Vex, the criterion used for denoising policy gradient updates. (b) We conduct experiments across a variety of benchmark environments, including standard continuous control problems. Our results show better performance with SAUNA. (c) We investigate why Vex provides a reliable assessment for the selection of samples that will positively impact learning. (d) We show how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlDenoisingSimilar Papers 제목 키워드 기반
QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult. Effective policy extraction requires exploiting the criti…
Reinforcement LearningInvestigation on the generalization of the Sampled Policy Gradient algorithm
The Sampled Policy Gradient (SPG) algorithm is a new offline actor-critic variant that samples in the action space to approximate the policy gradient. It does so by using the critic to evaluate the sampled actions. SPG o…
Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL
In reinforcement learning, policy gradient algorithms optimize the policy directly and rely on sampling efficiently an environment. Nevertheless, while most sampling procedures are based on direct policy sampling, self-p…
continuous-controlContinuous ControlDenoisingPolicy Gradient Methods+1Model-Based Policy Gradients with Parameter-Based Exploration by Least-Squares Conditional Density Estimation
The goal of reinforcement learning (RL) is to let an agent learn an optimal control policy in an unknown environment so that future expected rewards are maximized. The model-free RL approach directly learns the policy ba…
Density EstimationReinforcement LearningReinforcement Learning (RL)Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this en…
Multivariate Time Series ForecastingProbabilistic Time Series ForecastingTime SeriesTime Series Analysis+1