Parameter-free projected gradient descent

2023-05-31 · Evgenii Chzhen, Christophe Giraud, Gilles Stoltz

We consider the problem of minimizing a convex function over a closed convex set, with Projected Gradient Descent (PGD). We propose a fully parameter-free version of AdaGrad, which is adaptive to the distance between the initialization and the optimum, and to the sum of the square norm of the subgradients. Our algorithm is able to handle projection steps, does not involve restarts, reweighing along the trajectory or additional gradient evaluations compared to the classical PGD. It also fulfills optimal rates of convergence for cumulative regret up to logarithmic factors. We provide an extension of our approach to stochastic optimization and conduct numerical experiments supporting the developed theory.

📄 PDF Abstract BibTeX arXiv:2305.19605

Code (0)

등록된 구현이 없습니다.

Tasks

Stochastic Optimization

Methods 이 논문이 사용한 방법론

AdaGrad AdaGrad is a stochastic optimization method that adapts the learning rate to the parameters. It performs smaller updates for parameters associated with frequently occurring…

Deep Learning-Aided Projected Gradient Detector for Massive Overloaded MIMO Channels

2018-06-28 · Satoshi Takabe, Masayuki Imanishi, Tadashi Wadayama, Kazunori Hayashi

The paper presents a deep learning-aided iterative detection algorithm for massive overloaded MIMO systems. Since the proposed algorithm is based on the projected gradient descent method with trainable parameters, it is …

Deep Learning

Projected Wasserstein gradient descent for high-dimensional Bayesian inference

2021-02-12 · Yifei Wang, Peng Chen, Wuchen Li

We propose a projected Wasserstein gradient descent method (pWGD) for high-dimensional Bayesian inference problems. The underlying density function of a particle system of WGD is approximated by kernel density estimation…

Bayesian Inference Density Estimation Vocal Bursts Intensity Prediction

Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning

2026-01-19 · Ishir Garg, Neel Kolhe, Andy Peng, Rohan Gopalam arxiv

Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tas…

Continual Learning

Geodesic Gradient Descent: A Generic and Learning-rate-free Optimizer on Objective Function-induced Manifolds

2026-02-28 · Liwei Hu, Guangyao Li, Wenyong Wang, Xiaoming Zhang 외 arxiv

Euclidean gradient descent algorithms barely capture the geometry of objective function-induced hypersurfaces and risk driving update trajectories off the hypersurfaces. Riemannian gradient descent algorithms address the…

Projected Stein Variational Gradient Descent

2020-02-09 · NeurIPS 2020 12 · Peng Chen, Omar Ghattas

The curse of dimensionality is a longstanding challenge in Bayesian inference in high dimensions. In this work, we propose a projected Stein variational gradient descent (pSVGD) method to overcome this challenge by explo…

Bayesian Inference