Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.
Code (1)
Tasks
Meta Reinforcement LearningOff-policy evaluationreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. …
Atari GamesMeta Reinforcement Learningreinforcement-learningReinforcement Learning (RL)+1Acceleration in Policy Optimization
We work towards a unifying paradigm for accelerating policy optimization methods in reinforcement learning (RL) by integrating foresight in the policy improvement step via optimistic and adaptive updates. Leveraging the …
Meta-LearningPolicy Gradient MethodsReinforcement Learning (RL)Statistically Efficient Off-Policy Policy Gradients
Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value. In this paper, we consider the statistically efficient estimation of p…
Policy Gradient MethodsReinforcement LearningSmoothing Policies and Safe Policy Gradients
Policy Gradient (PG) algorithms are among the best candidates for the much-anticipated applications of reinforcement learning to real-world control tasks, such as robotics. However, the trial-and-error nature of these me…
Reinforcement LearningStochastic OptimizationSafe, Efficient, and Robust Reinforcement Learning for Ranking and Diffusion Models
This dissertation investigates how reinforcement learning (RL) methods can be designed to be safe, sample-efficient, and robust. Framed through the unifying perspective of contextual-bandit RL, the work addresses two maj…
Reinforcement Learning