Conditions for Convergence in Regularized Machine Learning Objectives
Analysis of the convergence rates of modern convex optimization algorithms can be achived through binary means: analysis of emperical convergence, or analysis of theoretical convergence. These two pathways of capturing information diverge in efficacy when moving to the world of distributed computing, due to the introduction of non-intuitive, non-linear slowdowns associated with broadcasting, and in some cases, gathering operations. Despite these nuances in the rates of convergence, we can still show the existence of convergence, and lower bounds for the rates. This paper will serve as a helpful cheat-sheet for machine learning practitioners encountering this problem class in the field.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningDistributed ComputingSimilar Papers 제목 키워드 기반
A Convergence Result for Regularized Actor-Critic Methods
In this paper, we present a probability one convergence proof, under suitable conditions, of a certain class of actor-critic algorithms for finding approximate solutions to entropy-regularized MDPs using the machinery of…
SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix
This paper introduces a new stochastic optimization method based on the regularized Fisher information matrix (FIM), named SOFIM, which can efficiently utilize the FIM to approximate the Hessian matrix for finding Newton…
image-classificationImage ClassificationStochastic OptimizationStochastic Primal-Dual Proximal ExtraGradient Descent for Compositely Regularized Optimization
We consider a wide range of regularized stochastic minimization problems with two regularization terms, one of which is composed with a linear function. This optimization model abstracts a number of important application…
regressionFast Rates for Regularized Objectives
We show that the empirical minimizer of a stochastic strongly convex objective, where the stochastic component is linear, converges to the population minimizer with rate $O(1/n)$. The result applies, in particular, to th…
RES: Regularized Stochastic BFGS Algorithm
RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with stochastic objectives. The use of stochastic gradient descen…
Second-order methods