Principled Curriculum Learning using Parameter Continuation Methods
In this work, we propose a parameter continuation method for the optimization of neural networks. There is a close connection between parameter continuation, homotopies, and curriculum learning. The methods we propose here are theoretically justified and practically effective for several problems in deep neural networks. In particular, we demonstrate better generalization performance than state-of-the-art optimization techniques such as ADAM for supervised and unsupervised learning tasks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Parameter Continuation Methods for the Optimization of Deep Neural Networks
There are many extant methods for approximating the solutions of non-convex optimization problems arising in deep neural networks, including stochastic gradient descent, RMSProp, AdaGrad, and ADAM. In this paper, we prop…
Mollifying Networks
The optimization of deep neural networks can be more challenging than traditional convex optimization problems due to the highly non-convex nature of the loss function, e.g. it can involve pathological landscapes such as…
Continuation methods as a tool for parameter inference in electrophysiology modeling
Parameterizing mathematical models of biological systems often requires fitting to stable periodic data. In cardiac electrophysiology this typically requires converging to a stable action potential through long simulatio…
FANS: Fast Non-Autoregressive Sequence Generation for Item List Continuation
User-curated item lists, such as video-based playlists on Youtube and book-based lists on Goodreads, have become prevalent for content sharing on online platforms. Item list continuation is proposed to model the overall …
Localized Physics-informed Gaussian Processes with Curriculum Training for Topology Optimization
We introduce a simultaneous and meshfree topology optimization (TO) framework based on physics-informed Gaussian processes (GPs). Our framework endows all design and state variables via GP priors which have a shared, mul…
Gaussian Processes