Online hyperparameter optimization by real-time recurrent learning
Conventional hyperparameter optimization methods are computationally intensive and hard to generalize to scenarios that require dynamically adapting hyperparameters, such as life-long learning. Here, we propose an online hyperparameter optimization algorithm that is asymptotically exact and computationally tractable, both theoretically and practically. Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in recurrent neural networks (RNNs). It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously, without repeatedly rolling out iterative optimization. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
Code (1)
Tasks
Hyperparameter OptimizationSimilar Papers 제목 키워드 기반
Time Series Kernels based on Nonlinear Vector AutoRegressive Delay Embeddings
Kernel design is a pivotal but challenging aspect of time series analysis, especially in the context of small datasets. In recent years, Reservoir Computing (RC) has emerged as a powerful tool to compare time series base…
Genetic algorithm-based hyperparameter optimization of deep learning models for PM2.5 time-series prediction
Since air pollution negatively affects human health and causes serious diseases, accurate air pollution prediction is essential regarding environmental sustainability. Although conventional statistical and machine learni…
Air Pollution PredictionDeep LearningHyperparameter OptimizationTime Series+2Efficient Online Hyperparameter Optimization for Kernel Ridge Regression with Applications to Traffic Time Series Prediction
Computational efficiency is an important consideration for deploying machine learning models for time series prediction in an online setting. Machine learning algorithms adjust model parameters automatically based on the…
BIG-bench Machine LearningComputational EfficiencyHyperparameter OptimizationPrediction+5Forward and Reverse Gradient-Based Hyperparameter Optimization
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. Thes…
Hyperparameter OptimizationOnline Natural Gradient as a Kalman Filter
We cast Amari's natural gradient in statistical learning as a specific case of Kalman filtering. Namely, applying an extended Kalman filter to estimate a fixed unknown parameter of a probabilistic model from a series of …