A Convergent ADMM Framework for Efficient Neural Network Training
As a well-known optimization framework, the Alternating Direction Method of Multipliers (ADMM) has achieved tremendous success in many classification and regression applications. Recently, it has attracted the attention of deep learning researchers and is considered to be a potential substitute to Gradient Descent (GD). However, as an emerging domain, several challenges remain unsolved, including 1) The lack of global convergence guarantees, 2) Slow convergence towards solutions, and 3) Cubic time complexity with regard to feature dimensions. In this paper, we propose a novel optimization framework to solve a general neural network training problem via ADMM (dlADMM) to address these challenges simultaneously. Specifically, the parameters in each layer are updated backward and then forward so that parameter information in each layer is exchanged efficiently. When the dlADMM is applied to specific architectures, the time complexity of subproblems is reduced from cubic to quadratic via a dedicated algorithm design utilizing quadratic approximations and backtracking techniques. Last but not least, we provide the first proof of convergence to a critical point sublinearly for an ADMM-type method (dlADMM) under mild conditions. Experiments on seven benchmark datasets demonstrate the convergence, efficiency, and effectiveness of our proposed dlADMM algorithm.
Code (1)
Tasks
Efficient Neural NetworkMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
P-ADMMiRNN: Training RNN with Stable Convergence via An Efficient and Paralleled ADMM Approach
It is hard to train Recurrent Neural Network (RNN) with stable convergence and avoid gradient vanishing and exploding problems, as the weights in the recurrent unit are repeated from iteration to iteration. Moreover, RNN…
text-classificationText ClassificationConvergent ADMM Plug and Play PET Image Reconstruction
In this work, we investigate hybrid PET reconstruction algorithms based on coupling a model-based variational reconstruction and the application of a separately learnt Deep Neural Network operator (DNN) in an ADMM Plug a…
Image ReconstructionProvably Convergent Plug & Play Linearized ADMM, applied to Deblurring Spatially Varying Kernels
Plug & Play methods combine proximal algorithms with denoiser priors to solve inverse problems. These methods rely on the computability of the proximal operator of the data fidelity term. In this paper, we propose a Plug…
DeblurringSuper-ResolutionTaming Score-Based Denoisers in ADMM: A Convergent Plug-and-Play Framework
While score-based generative models have emerged as powerful priors for solving inverse problems, directly integrating them into optimization algorithms such as ADMM remains nontrivial. Two central challenges arise: i) t…
Convergence Analysis of Consensus-ADMM for General QCQP
We analyze the convergence properties of the consensus-alternating direction method of multipliers (ADMM) for solving general quadratically constrained quadratic programs. We prove that the augmented Lagrangian function …