Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks
In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two key mechanisms: one involves building a homotopy activation function that seamlessly connects the linear activation function with the ReLU activation function; the other technique entails relaxing the homotopy parameter to enhance the training refinement process. We have conducted an in-depth analysis of this novel method within the context of the neural tangent kernel (NTK), revealing significantly improved convergence rates. Our experimental results, especially when considering networks with larger widths, validate the theoretical conclusions. This proposed HRTA exhibits the potential for other activation functions and deep neural networks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Homotopy relations for digital images
We introduce three generalizations of homotopy equivalence in digital images, to allow us to express whether a finite and an infinite digital image are similar with respect to homotopy. We show that these three general…
Homotopy based algorithms for $\ell_0$-regularized least-squares
Sparse signal restoration is usually formulated as the minimization of a quadratic cost function $\|y-Ax\|_2^2$, where A is a dictionary and x is an unknown sparse vector. It is well-known that imposing an $\ell_0$ const…
Heuristic SearchConvex Formulations for Training Two-Layer ReLU Neural Networks
Solving non-convex, NP-hard optimization problems is crucial for training machine learning models, including neural networks. However, non-convexity often leads to black-box machine learning models with unclear inner wor…
Continuation Path Learning for Homotopy Optimization
Homotopy optimization is a traditional method to deal with a complicated optimization problem by solving a sequence of easy-to-hard surrogate subproblems. However, this method can be very sensitive to the continuation sc…
Decision MakingA conditional gradient homotopy method with applications to Semidefinite Programming
We propose a new homotopy-based conditional gradient method for solving convex optimization problems with a large number of simple conic constraints. Instances of this template naturally appear in semidefinite programmin…
Combinatorial Optimization