A Momentum Accelerated Adaptive Cubic Regularization Method for Nonconvex Optimization

2022-10-12 · Yihang Gao, Michael K. Ng

The cubic regularization method (CR) and its adaptive version (ARC) are popular Newton-type methods in solving unconstrained non-convex optimization problems, due to its global convergence to local minima under mild conditions. The main aim of this paper is to develop a momentum-accelerated adaptive cubic regularization method (ARCm) to improve the convergent performance. With the proper choice of momentum step size, we show the global convergence of ARCm and the local convergence can also be guaranteed under the \KL property. Such global and local convergence can also be established when inexact solvers with low computational costs are employed in the iteration procedure. Numerical results for non-convex logistic regression and robust linear regression models are reported to demonstrate that the proposed ARCm significantly outperforms state-of-the-art cubic regularization methods (e.g., CR, momentum-based CR, ARC) and the trust region method. In particular, the number of iterations required by ARCm is less than 10\% to 50\% required by the most competitive method (ARC) in the experiments.

📄 PDF Abstract BibTeX arXiv:2210.05987

Code (0)

등록된 구현이 없습니다.

Tasks

ARCregression

Methods 이 논문이 사용한 방법론

Logistic Regression Logistic Regression, despite its name, is a linear model for classification rather than regression. Logistic regression is also known in the literature as logit regression,…

Linear Regression Linear Regression is a method for modelling a relationship between a dependent variable and independent variables. These models can be fit with numerous approaches. The most…

Cubic Regularization with Momentum for Nonconvex Optimization

2018-10-09 · Zhe Wang, Yi Zhou, Yingbin Liang, Guanghui Lan

Momentum is a popular technique to accelerate the convergence in practical training, and its impact on convergence guarantee has been well-studied for first-order algorithms. However, such a successful acceleration techn…

Accelerated Block Coordinate Proximal Gradients with Applications in High Dimensional Statistics

2017-10-15 · Tsz Kit Lau, Yuan YAO

Nonconvex optimization problems arise in different research fields and arouse lots of attention in signal processing, statistics and machine learning. In this work, we explore the accelerated proximal gradient method and…

BIG-bench Machine Learning regression Vocal Bursts Intensity Prediction

Sharp Analysis for Nonconvex SGD Escaping from Saddle Points

2019-02-01 · Cong Fang, Zhouchen Lin, Tong Zhang

In this paper, we give a sharp analysis for Stochastic Gradient Descent (SGD) and prove that SGD is able to efficiently escape from saddle points and find an $(\epsilon, O(\epsilon^{0.5}))$-approximate second-order stati…

Stochastic Optimization

Acceleration and Implicit Regularization in Gaussian Phase Retrieval

2023-11-21 · Tyler Maunu, Martin Molina-Fructuoso

We study accelerated optimization methods in the Gaussian phase retrieval problem. In this setting, we prove that gradient methods with Polyak or Nesterov momentum have similar implicit regularization to gradient descent…

Retrieval

Combining Stochastic Adaptive Cubic Regularization with Negative Curvature for Nonconvex Optimization

2019-06-27 · Seonho Park, Seung Hyun Jung, Panos M. Pardalos

We focus on minimizing nonconvex finite-sum functions that typically arise in machine learning problems. In an attempt to solve this problem, the adaptive cubic regularized Newton method has shown its strong global conve…

BIG-bench Machine Learning