A Non-Monotone Preconditioned Trust-Region Method for Neural Network Training
Training deep neural networks at scale can benefit from domain decomposition, where the network is split into subdomains trained in parallel and coupled by a global trust-region mechanism. Building on the Additively Preconditioned Trust-Region Strategy (APTS), we propose a non-monotone variant with a nonlinear additive Schwarz preconditioner that combines parallel subdomain corrections with global coarse-space directions. A windowed acceptance criterion allows controlled objective increases, avoiding needless rejection of effective coarse steps. The resulting non-monotone APTS (NAPTS) preserves accuracy while reducing CPU time by 30\% and cutting rejected steps to one third of those in APTS.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Data-Parallel Neural Network Training via Nonlinearly Preconditioned Trust-Region Method
Parallel training methods are increasingly relevant in machine learning (ML) due to the continuing growth in model and dataset sizes. We propose a variant of the Additively Preconditioned Trust-Region Strategy (APTS) for…
Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods
We propose to train neural networks (NNs) using a novel variant of the ``Additively Preconditioned Trust-region Strategy'' (APTS). The proposed method is based on a parallelizable additive domain decomposition approach a…
A Neural-Operator Preconditioned Newton Method for Accelerated Nonlinear Solvers
We propose a novel neural preconditioned Newton (NP-Newton) method for solving parametric nonlinear systems of equations. To overcome the stagnation or instability of Newton iterations caused by unbalanced nonlinearities…
Computational EfficiencyStochastic Trust Region Inexact Newton Method for Large-scale Machine Learning
Nowadays stochastic approximation methods are one of the major research direction to deal with the large-scale machine learning problems. From stochastic first order methods, now the focus is shifting to stochastic secon…
BIG-bench Machine LearningSecond-order methodsSpingarn's Method and Progressive Decoupling Beyond Elicitable Monotonicity
Spingarn's method of partial inverses and the progressive decoupling algorithm address inclusion problems involving the sum of an operator and the normal cone of a linear subspace, known as linkage problems. Despite thei…