Towards Hyperparameter-Agnostic DNN Training via Dynamical System Insights
We present a stochastic first-order optimization method specialized for deep neural networks (DNNs), ECCO-DNN. This method models the optimization variable trajectory as a dynamical system and develops a discretization algorithm that adaptively selects step sizes based on the trajectory's shape. This provides two key insights: designing the dynamical system for fast continuous-time convergence and developing a time-stepping algorithm to adaptively select step sizes based on principles of numerical integration and neural network structure. The result is an optimizer with performance that is insensitive to hyperparameter variations and that achieves comparable performance to state-of-the-art optimizers including ADAM, SGD, RMSProp, and AdaGrad. We demonstrate this in training DNN models and datasets, including CIFAR-10 and CIFAR-100 using ECCO-DNN and find that ECCO-DNN's single hyperparameter can be changed by three orders of magnitude without affecting the trained models' accuracies. ECCO-DNN's insensitivity reduces the data and computation needed for hyperparameter tuning, making it advantageous for rapid prototyping and for applications with new datasets. To validate the efficacy of our proposed optimizer, we train an LSTM architecture on a household power consumption dataset with ECCO-DNN and achieve an optimal mean-square-error without tuning hyperparameters.
Code (0)
등록된 구현이 없습니다.
Tasks
Numerical IntegrationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
AutoRL Hyperparameter Landscapes
Although Reinforcement Learning (RL) has shown to be capable of producing impressive results, its use is limited by the impact of its hyperparameters on performance. This often makes it difficult to achieve good results …
AutoMLHyperparameter OptimizationOpen-Ended Question AnsweringReinforcement Learning (RL)On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
Model-based Reinforcement Learning (MBRL) is a promising framework for learning control in a data-efficient manner. MBRL algorithms can be fairly complex due to the separate dynamics modeling and the subsequent planning …
Hyperparameter OptimizationModel-based Reinforcement Learningreinforcement-learningReinforcement Learning (RL)HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks
To mitigate the pain of manually tuning hyperparameters of deep neural networks, automated machine learning (AutoML) methods have been developed to search for an optimal set of hyperparameters in large combinatorial sear…
AutoMLHyperparameter OptimizationHyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks
We introduce Hyperparameter Controller (HyperController), a computationally efficient algorithm for hyperparameter optimization during training of reinforcement learning neural networks. HyperController optimizes hyperpa…
Hyperparameter Optimizationreinforcement-learningReinforcement LearningIterated Population Based Training with Task-Agnostic Restarts
Hyperparameter Optimization (HPO) can lift the burden of tuning hyperparameters (HPs) of neural networks. HPO algorithms from the Population Based Training (PBT) family are efficient thanks to dynamically adjusting HPs e…
Hyperparameter OptimizationReinforcement LearningImage Classification