AN EFFICIENT HOMOTOPY TRAINING ALGORITHM FOR NEURAL NETWORKS

2020-01-01 · ICLR 2020 1 · Qipin Chen, Wenrui Hao

We present a Homotopy Training Algorithm (HTA) to solve optimization problems arising from neural networks. The HTA starts with several decoupled systems with low dimensional structure and tracks the solution to the high dimensional coupled system. The decoupled systems are easy to solve due to the low dimensionality but can be connected to the original system via a continuous homotopy path guided by the HTA. We have proved the convergence of HTA for the non-convex case and existence of the homotopy solution path for the convex case. The HTA has provided a better accuracy on several examples including VGG models on CIFAR-10. Moreover, the HTA would be combined with the dropout technique to provide an alternative way to train the neural networks.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Max Pooling Max Pooling is a pooling operation that calculates the maximum value for patches of a feature map, and uses it to create a downsampled (pooled) feature map. It is usually…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Ethereum Customer Service Number +1-833-534-1729 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks

2023-09-26 · Yahong Yang, Qipin Chen, Wenrui Hao

In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two …

Homotopy Analysis for Tensor PCA

2016-10-28 · Anima Anandkumar, Yuan Deng, Rong Ge, Hossein Mobahi

Developing efficient and guaranteed nonconvex algorithms has been an important challenge in modern machine learning. Algorithms with good empirical performance such as stochastic gradient descent often lack theoretical g…

global-optimization

Convergence Analysis of Homotopy-SGD for non-convex optimization

2020-11-20 · Matilde Gargiani, Andrea Zanelli, Quoc Tran-Dinh, Moritz Diehl 외

First-order stochastic methods for solving large-scale non-convex optimization problems are widely used in many big-data applications, e.g. training deep neural networks as well as other complex and potentially non-conve…

HUNet: Homotopy Unfolding Network for Image Compressive Sensing

2025-01-01 · CVPR 2025 1 · Feiyang Shen, Hongping Gan

Deep Unfolding Networks (DUNs) have risen to prominence due to their interpretability and superior performance for image Compressive Sensing (CS). However, existing DUNs still face significant issues, such as the ins…

Compressive Sensing Image Reconstruction

A Homotopy Algorithm for Optimal Transport

2021-12-13 · Roozbeh Yousefzadeh

The optimal transport problem has many applications in machine learning, physics, biology, economics, etc. Although its goal is very clear and mathematically well-defined, finding its optimal solution can be challenging …

Form