Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information
We present a novel adaptive optimization algorithm for large-scale machine learning problems. Equipped with a low-cost estimate of local curvature and Lipschitz smoothness, our method dynamically adapts the search direction and step-size. The search direction contains gradient information preconditioned by a well-scaled diagonal preconditioning matrix that captures the local curvature information. Our methodology does not require the tedious task of learning rate tuning, as the learning rate is updated automatically without adding an extra hyperparameter. We provide convergence guarantees on a comprehensive collection of optimization problems, including convex, strongly convex, and nonconvex problems, in both deterministic and stochastic regimes. We also conduct an extensive empirical evaluation on standard machine learning problems, justifying our algorithm's versatility and demonstrating its strong performance compared to other start-of-the-art first-order and second-order methods.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningSecond-order methodsSimilar Papers 제목 키워드 기반
Doubly Stochastic Adaptive Neighbors Clustering via the Marcus Mapping
Clustering is a fundamental task in machine learning and data science, and similarity graph-based clustering is an important approach within this domain. Doubly stochastic symmetric similarity graphs provide numerous ben…
ClusteringComputational EfficiencyDAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning
Federated Learning (FL) is a powerful technique for training a model on a server with data from several clients in a privacy-preserving manner. In FL, a server sends the model to every client, who then train the model lo…
Federated LearningPrivacy PreservingQuantizationStochastic Gradient Methods with Preconditioned Updates
This work considers the non-convex finite sum minimization problem. There are several algorithms for such problems, but existing methods often work poorly when the problem is badly scaled and/or ill-conditioned, and a pr…
Optimization for Large-Scale Machine Learning with Distributed Features and Observations
As the size of modern data sets exceeds the disk and memory capacities of a single computer, machine learning practitioners have resorted to parallel and distributed computing. Given that optimization is one of the pilla…
BIG-bench Machine LearningDistributed ComputingDistributed OptimizationDoubly Robust Adaptive Conformal Inference for Causal Effects Under Temporal Dependence
We propose doubly robust adaptive conformal inference (DR-ACI), which constructs prediction intervals for doubly robust pseudo-outcomes under temporal dependence.