Generalized Advantage Estimation for Distributional Policy Gradients
Generalized Advantage Estimation (GAE) has been used to mitigate the computational complexity of reinforcement learning (RL) by employing an exponentially weighted estimation of the advantage function to reduce the variance in policy gradient estimates. Despite its effectiveness, GAE is not designed to handle value distributions integral to distributional RL, which can capture the inherent stochasticity in systems and is hence more robust to system noises. To address this gap, we propose a novel approach that utilizes the optimal transport theory to introduce a Wasserstein-like directional metric, which measures both the distance and the directional discrepancies between probability distributions. Using the exponentially weighted estimation, we leverage this Wasserstein-like directional metric to derive distributional GAE (DGAE). Similar to traditional GAE, our proposed DGAE provides a low-variance advantage estimate with controlled bias, making it well-suited for policy gradient algorithms that rely on advantage estimation for policy updates. We integrated DGAE into three different policy gradient methods. Algorithms were evaluated across various OpenAI Gym environments and compared with the baselines with traditional GAE to assess the performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningOpenAI GymSimilar Papers 제목 키워드 기반
An Approximate Ascent Approach To Prove Convergence of PPO
Proximal Policy Optimization (PPO) is among the most widely used deep reinforcement learning algorithms, yet its theoretical foundations remain incomplete. Most importantly, convergence and understanding of fundamental P…
Reinforcement LearningValue Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradients
It is vital to accurately estimate the value function in Deep Reinforcement Learning (DRL) such that the agent could execute proper actions instead of suboptimal ones. However, existing actor-critic methods suffer more o…
continuous-controlContinuous ControlDeep Reinforcement LearningBootstrap Advantage Estimation for Policy Optimization in Reinforcement Learning
This paper proposes an advantage estimation approach based on data augmentation for policy optimization. Unlike using data augmentation on the input to learn value and policy function as existing methods use, our method …
Data Augmentationreinforcement-learningReinforcement LearningReinforcement Learning (RL)Bayesian Distributional Policy Gradients
Distributional Reinforcement Learning (RL) maintains the entire probability distribution of the reward-to-go, i.e. the return, providing more learning signals that account for the uncertainty associated with policy perfo…
Atari GamesContrastive LearningDistributional Reinforcement LearningMuJoCo+1Fast moment estimation for generalized latent Dirichlet models
We develop a generalized method of moments (GMM) approach for fast parameter estimation in a new class of Dirichlet latent variable models with mixed data types. Parameter estimation via GMM has been demonstrated to have…
parameter estimationVariational Inference