AGGLIO: Global Optimization for Locally Convex Functions

2021-11-06 · Debojyoti Dey, Bhaskar Mukhoty, Purushottam Kar

This paper presents AGGLIO (Accelerated Graduated Generalized LInear-model Optimization), a stage-wise, graduated optimization technique that offers global convergence guarantees for non-convex optimization problems whose objectives offer only local convexity and may fail to be even quasi-convex at a global scale. In particular, this includes learning problems that utilize popular activation functions such as sigmoid, softplus and SiLU that yield non-convex training objectives. AGGLIO can be readily implemented using point as well as mini-batch SGD updates and offers provable convergence to the global optimum in general conditions. In experiments, AGGLIO outperformed several recently proposed optimization techniques for non-convex and locally convex objectives in terms of convergence rate as well as convergent accuracy. AGGLIO relies on a graduation technique for generalized linear models, as well as a novel proof strategy, both of which may be of independent interest.

📄 PDF Abstract BibTeX arXiv:2111.03932

Code (1)

purushottamkar/agglio 공식 구현

Tasks

global-optimizationModel Optimization

Methods 이 논문이 사용한 방법론

SiLU Sigmoid Linear Units, or SiLUs, are activation functions for neural networks. The activation of the SiLU is computed by the sigmoid function multiplied by its input, or…

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Beyond Convexity: Stochastic Quasi-Convex Optimization

2015-07-08 · NeurIPS 2015 12 · Elad Hazan, Kfir. Y. Levy, Shai Shalev-Shwartz

Stochastic convex optimization is a basic and well studied primitive in machine learning. It is well known that convex and Lipschitz functions can be minimized efficiently using Stochastic Gradient Descent (SGD). The Nor…

Revisiting Projection-Free Optimization for Strongly Convex Constraint Sets

2018-11-14 · Jarrid Rector-Brooks, Jun-Kun Wang, Barzan Mozafari

We revisit the Frank-Wolfe (FW) optimization under strongly convex constraint sets. We provide a faster convergence rate for FW without line search, showing that a previously overlooked variant of FW is indeed faster tha…

Locally Convex Global Loss Network for Decision-Focused Learning

2024-03-04 · Haeun Jeon, Hyunglip Bae, Minsu Park, Chanyeong Kim 외

In decision-making problems under uncertainty, predicting unknown parameters is often considered independent of the optimization part. Decision-focused learning (DFL) is a task-oriented framework that integrates predicti…

Decision Making

Coordinate Descent Methods for DC Minimization: Optimality Conditions and Global Convergence

2021-09-09 · Ganzhao Yuan

Difference-of-Convex (DC) minimization, referring to the problem of minimizing the difference of two convex functions, has been found rich applications in statistical learning and studied extensively for decades. However…

Convergent Policy Optimization for Safe Reinforcement Learning

2019-10-26 · NeurIPS 2019 12 · Ming Yu, Zhuoran Yang, Mladen Kolar, Zhaoran Wang

We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonco…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1