Online Hyperparameter Adaptation via Amortized Proximal Optimization

2019-05-01 · ICLR 2019 5 · Paul Vicol, Jeffery Z. HaoChen, Roger Grosse

Effective performance of neural networks depends critically on effective tuning of optimization hyperparameters, especially learning rates (and schedules thereof). We present Amortized Proximal Optimization (APO), which takes the perspective that each optimization step should approximately minimize a proximal objective (similar to the ones used to motivate natural gradient and trust region policy optimization). Optimization hyperparameters are adapted to best minimize the proximal objective after one weight update. We show that an idealized version of APO (where an oracle minimizes the proximal objective exactly) achieves global convergence to stationary point and locally second-order convergence to global optimum for neural networks. APO incurs minimal computational overhead. We experiment with using APO to adapt a variety of optimization hyperparameters online during training, including (possibly layer-specific) learning rates, damping coefficients, and gradient variance exponents. For a variety of network architectures and optimization algorithms (including SGD, RMSprop, and K-FAC), we show that with minimal tuning, APO performs competitively with carefully tuned optimizers.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Amortized Proximal Optimization

2022-02-28 · Juhan Bae, Paul Vicol, Jeff Z. HaoChen, Roger Grosse

We propose a framework for online meta-optimization of parameters that govern optimization, called Amortized Proximal Optimization (APO). We first interpret various existing neural network optimizers as approximate stoch…

image-classification Image Classification Image Reconstruction Meta-Learning+1

Online Hyperparameter Search Interleaved with Proximal Parameter Updates

2020-04-06 · Luis Miguel Lopez-Ramos, Baltasar Beferull-Lozano

There is a clear need for efficient algorithms to tune hyperparameters for statistical learning schemes, since the commonly applied search methods (such as grid search with N-fold cross-validation) are inefficient and/or…

Hyperparameter Optimization

PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation

2018-10-05 · Perttu Hämäläinen, Amin Babadi, Xiaoxiao Ma, Jaakko Lehtinen

Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach. However, we observe that in a continuous action space, PPO can prematurely shrink the exploration variance, which le…

continuous-control Continuous Control Reinforcement Learning Reinforcement Learning (RL)

Hot Swapping for Online Adaptation of Optimization Hyperparameters

2014-12-20 · Kevin Bache, Dennis Decoste, Padhraic Smyth

We describe a general framework for online adaptation of optimization hyperparameters by `hot swapping' their values during learning. We investigate this approach in the context of adaptive learning rate selection using …

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

2024-02-22 · Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee 외

AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. Proximal Policy Optimization (PPO) has been posit…