Fast Efficient Hyperparameter Tuning for Policy Gradient Methods
The performance of policy gradient methods is sensitive to hyperparameter settings that must be tuned for any new application. Widely used grid search methods for tuning hyperparameters are sample inefficient and computationally expensive. More advanced methods like Population Based Training that learn optimal schedules for hyperparameters instead of fixed settings can yield better results, but are also sample inefficient and computationally expensive. In this paper, we propose Hyperparameter Optimisation on the Fly (HOOF), a gradient-free algorithm that requires no more than one training run to automatically adapt the hyperparameter that affect the policy update directly through the gradient. The main idea is to use existing trajectories sampled by the policy gradient method to optimise a one-step improvement objective, yielding a sample and computationally efficient algorithm that is easy to implement. Our experimental results across multiple domains and algorithms show that using HOOF to learn these hyperparameter schedules leads to faster learning with improved performance.
Code (1)
Tasks
Policy Gradient MethodsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fast Efficient Hyperparameter Tuning for Policy Gradients
The performance of policy gradient methods is sensitive to hyperparameter settings that must be tuned for any new application. Widely used grid search methods for tuning hyperparameters are sample inefficient and computa…
Meta-LearningPolicy Gradient MethodsAdaptive Diffusion Policy Optimization for Robotic Manipulation
Recent studies have shown the great potential of diffusion models in improving reinforcement learning (RL) by modeling complex policies, expressing a high degree of multi-modality, and efficiently handling high-dimension…
continuous-controlContinuous ControlReinforcement Learning (RL)Joint Descent: Training and Tuning Simultaneously
Typically in machine learning, training and tuning are done in an alternating manner: for a fixed set of hyperparameters $y$, we apply gradient descent to our objective $f(x, y)$ over trainable variables $x$ until conver…
Episodic Policy Gradient Training
We introduce a novel training procedure for policy gradient methods wherein episodic memory is used to optimize the hyperparameters of reinforcement learning algorithms on-the-fly. Unlike other hyperparameter searches, w…
Policy Gradient MethodsSchedulingCPMLHO:Hyperparameter Tuning via Cutting Plane and Mixed-Level Optimization
The hyperparameter optimization of neural network can be expressed as a bilevel optimization problem. The bilevel optimization is used to automatically update the hyperparameter, and the gradient of the hyperparameter is…
Bilevel OptimizationHyperparameter Optimization