Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

2023-07-19 · Nachuan Xiao, Xiaoyin Hu, Kim-Chuan Toh

In this paper, we focus on providing convergence guarantees for stochastic subgradient methods in minimizing nonsmooth nonconvex functions. We first investigate the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function. We prove that, for any sequence of sufficiently small stepsizes and approximation parameters, coupled with sufficiently controlled noises, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion. Moreover, we develop an improved analysis to apply our proposed framework to establish the global stability of a wide range of stochastic subgradient methods, where the corresponding Lyapunov functions are possibly non-coercive. These theoretical results illustrate the promising potential of our proposed framework for establishing the global stability of various stochastic subgradient methods.

📄 PDF Abstract BibTeX arXiv:2307.10053

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Focus 설명 없음

Lion The Lion optimizer is discovered by symbolic program search. It is more memory-efficient than most adaptive optimizers as it only needs to momentum. The update of Lion is produced…

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Convergence Rates for Deterministic and Stochastic Subgradient Methods Without Lipschitz Continuity

2017-12-12 · Benjamin Grimmer

We extend the classic convergence rate theory for subgradient methods to apply to non-Lipschitz functions. For the deterministic projected subgradient method, we present a global $O(1/\sqrt{T})$ convergence rate for any …

Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition

2016-07-04 · Yi Xu, Qihang Lin, Tianbao Yang

In this paper, a new theory is developed for first-order stochastic convex optimization, showing that the global convergence rate is sufficiently quantified by a local growth rate of the objective function in a neighborh…

Stochastic Optimization

Decentralized Stochastic Subgradient-type Methods with Communication Compression for Nonsmooth Nonconvex Optimization

2026-07-02 · Siyuan Zhang, Nachuan Xiao, Xin Liu arxiv

In this paper, we consider the nonsmooth nonconvex decentralized optimization problem, where inter-agent communication is compressed. We propose a general framework that unifies various decentralized stochastic subgradie…

On Distributed Non-convex Optimization: Projected Subgradient Method For Weakly Convex Problems in Networks

2020-04-28 · Shixiang Chen, Alfredo Garcia, Shahin Shahrampour

The stochastic subgradient method is a widely-used algorithm for solving large-scale optimization problems arising in machine learning. Often these problems are neither smooth nor convex. Recently, Davis et al. [1-2] cha…

Compressive Sensing Dictionary Learning Retrieval

Stochastic Convex Optimization: Faster Local Growth Implies Faster Global Convergence

2017-08-01 · ICML 2017 8 · Yi Xu, Qihang Lin, Tianbao Yang