Papers Second-order methods
“Second-order methods” 태그가 달린 논문 181편 · 필터 해제
NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation
Adaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization. In contrast, second-order methods enhance convergence and generalization but typically incur h…
Second-order methodsKOALA++: Efficient Kalman-Based Optimization of Neural Networks with Gradient-Covariance Products
We propose KOALA++, a scalable Kalman-based optimization algorithm that explicitly models structured gradient uncertainty in neural network training. Unlike second-order methods, which rely on expensive second order grad…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+1Accelerated Training of Federated Learning via Second-Order Methods
This paper explores second-order optimization methods in Federated Learning (FL), addressing the critical challenges of slow convergence and the excessive communication rounds required to achieve optimal performance from…
Federated LearningSecond-order methodsTowards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis
First-order optimization methods remain the standard for training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by preconditioning the stochastic gradient with a diagonal mat…
Computational Efficiencyimage-classificationImage ClassificationLanguage Modeling+2Representation Meets Optimization: Training PINNs and PIKANs for Gray-Box Discovery in Systems Pharmacology
Physics-Informed Kolmogorov-Arnold Networks (PIKANs) are gaining attention as an effective counterpart to the original multilayer perceptron-based Physics-Informed Neural Networks (PINNs). Both representation models can …
Computational EfficiencyKolmogorov-Arnold NetworksSecond-order methodsFedOSAA: Improving Federated Learning with One-Step Anderson Acceleration
Federated learning (FL) is a distributed machine learning approach that enables multiple local clients and a central server to collaboratively train a model while keeping the data on their own devices. First-order method…
Federated LearningSecond-order methodsSASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. In this work, we look into this issue through the lens of the loss landscape and find that existing se…
Second-order methodsOnline Covariance Matrix Estimation in Sketched Newton Methods
Given the ubiquity of streaming data, online algorithms have been widely used for parameter estimation, with second-order methods particularly standing out for their efficiency and robustness. In this paper, we study an …
parameter estimationSecond-order methodsA Survey of Optimization Methods for Training DL Models: Theoretical Perspective on Convergence and Generalization
As data sets grow in size and complexity, it is becoming more difficult to pull useful features from them using hand-crafted feature extractors. For this reason, deep learning (DL) frameworks are now widely popular. The …
Distributed OptimizationNavigateSecond-order methodsDistributed Quasi-Newton Method for Fair and Fast Federated Learning
Federated learning (FL) is a promising technology that enables edge devices/clients to collaboratively and iteratively train a machine learning model under the coordination of a central server. The most common approach t…
FairnessFederated LearningSecond-order methodsPreconditioners for the Stochastic Training of Neural Fields
Neural fields encode continuous multidimensional signals as neural networks, enabling diverse applications in computer vision, robotics, and geometry. While Adam is effective for stochastic optimization, it often req…
Image ReconstructionNeRFSecond-order methodsStochastic OptimizationGP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning
Second-order methods are widely adopted to improve the convergence rate of learning algorithms. In federated learning (FL), these methods require the clients to share their local Hessian matrices with the parameter serve…
Federated LearningSecond-order methodsA Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methods
Tuning step sizes is crucial for the stability and efficiency of optimization algorithms. While adaptive coordinate-wise step sizes have been shown to outperform scalar step size in first-order methods, their use in seco…
Second-order methodsGradient Norm Regularization Second-Order Algorithms for Solving Nonconvex-Strongly Concave Minimax Problems
In this paper, we study second-order algorithms for solving nonconvex-strongly concave minimax problems, which have attracted much attention in recent years in many fields, especially in machine learning.We propose a gra…
Second-order methodsDon't Be So Positive: Negative Step Sizes in Second-Order Methods
The value of second-order methods lies in the use of curvature information. Yet, this information is costly to extract and once obtained, valuable negative curvature information is often discarded so that the method is g…
Second-order methodsExact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization
Second-order optimization has been shown to accelerate the training of deep neural networks in many applications, often yielding faster progress per iteration on the training loss compared to first-order optimizers. Howe…
Second-order methodsImproving Stochastic Cubic Newton with Momentum
We study stochastic second-order methods for solving general non-convex optimization problems. We propose using a special version of momentum to stabilize the stochastic gradient and Hessian estimates in Newton's method.…
Second-order methodsSecond-Order Min-Max Optimization with Lazy Hessians
This paper studies second-order methods for convex-concave minimax optimization. Monteiro and Svaiter (2012) proposed a method to solve the problem with an optimal iteration complexity of $\mathcal{O}(\epsilon^{-3/2})$ t…
Second-order methodsUnlocking FedNL: Self-Contained Compute-Optimized Implementation
Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner, eliminating the need for sharing their local data. The recent…
Federated LearningSecond-order methodsAdversarial Vulnerability as a Consequence of On-Manifold Inseparibility
Recent works have shown theoretically and empirically that redundant data dimensions are a source of adversarial vulnerability. However, the inverse doesn't seem to hold in practice; employing dimension-reduction techniq…
AttributeDimensionality ReductionSecond-order methods