Identifying Policy Gradient Subspaces
Policy gradient methods hold great potential for solving complex continuous control tasks. Still, their training efficiency can be improved by exploiting structure within the optimization problem. Recent work indicates that supervised learning can be accelerated by leveraging the fact that gradients lie in a low-dimensional and slowly-changing subspace. In this paper, we conduct a thorough evaluation of this phenomenon for two popular deep policy gradient methods on various simulated benchmark tasks. Our results demonstrate the existence of such gradient subspaces despite the continuously changing data distribution inherent to reinforcement learning. These findings reveal promising directions for future work on more efficient reinforcement learning, e.g., through improving parameter-space exploration or enabling second-order optimization.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlPolicy Gradient Methodsreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
PROMA: Projected Microbatch Accumulation for Reference-Free Proximal Policy Updates
This note introduces Projected Microbatch Accumulation (PROMA), a reference-free proximal policy method that controls KL divergence by projecting away high-variance components of the policy gradient. Two variants are pre…
Reinforcement Learning with Subspaces using Free Energy Paradigm
In large-scale problems, standard reinforcement learning algorithms suffer from slow learning speed. In this paper, we follow the framework of using subspaces to tackle this problem. We propose a free-energy minimization…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Thompson SamplingVariance Reduced Domain Randomization for Policy Gradient
By introducing randomness on environment parameters that fundamentally affect the dynamics, domain randomization (DR) imposes diversity to the policy trained by deep reinforcement learning, and thus improves its capabili…
Deep Reinforcement LearningPolicy Gradient MethodsExploiting generalization in the subspaces for faster model-based learning
Due to the lack of enough generalization in the state-space, common methods in Reinforcement Learning (RL) suffer from slow learning speed especially in the early learning trials. This paper introduces a model-based meth…
Decision MakingReinforcement LearningReinforcement Learning (RL)From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization
We present a new algorithm ASEBO for optimizing high-dimensional blackbox functions. ASEBO adapts to the geometry of the function and learns optimal sets of sensing directions, which are used to probe it, on-the-fly. It …
compressed sensingMulti-Armed BanditsReinforcement Learning