A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm

2026-05-28 · Sakshi Kumari, Shyam Kumar M, Sushmitha P arxiv

A crucial component of machine learning algorithms is minimizing loss functions with less computational cost and less oscillations. While adaptive learning rate-based optimizers have been widely used for real-world tasks, they do not guarantee convergence, which is why AMSGrad was later introduced to investigate the non-convergence behaviour of Adam. In this paper, popular adaptive optimization methods like Adam and AMSGrad are critically reviewed with an emphasis on their fundamental design concepts. To address limitations of the above mentioned optimizers, a new optimizer variant, C-Adam, is proposed based on the line of sight approach. A theoretical proof for convergence is also provided and the optimizer is validated through a number of real-life based numerical experiments.

📄 PDF Abstract BibTeX arXiv:2605.29273

Code (0)

등록된 구현이 없습니다.

Towards Self-adaptive Mutation in Evolutionary Multi-Objective Algorithms

2023-03-08 · Furong Ye, Frank Neumann, Jacob de Nobel, Aneta Neumann 외

Parameter control has succeeded in accelerating the convergence process of evolutionary algorithms. While empirical and theoretical studies have shed light on the behavior of algorithms for single-objective optimization,…

Benchmarking Evolutionary Algorithms

On Regret with Multiple Best Arms

2020-06-26 · NeurIPS 2020 12 · Yinglun Zhu, Robert Nowak

We study a regret minimization problem with the existence of multiple best/near-optimal arms in the multi-armed bandit setting. We consider the case when the number of arms/actions is comparable or much larger than the t…

Learning Data-adaptive Nonparametric Kernels

2018-08-31 · Fanghui Liu, Xiaolin Huang, Chen Gong, Jie Yang 외

In this paper, we propose a data-adaptive non-parametric kernel learning framework in margin based kernel methods. In model formulation, given an initial kernel matrix, a data-adaptive matrix with two constraints is impo…

Faster Adaptive Federated Learning

2022-12-02 · Xidong Wu, Feihu Huang, Zhengmian Hu, Heng Huang

Federated learning has attracted increasing attention with the emergence of distributed data. While extensive federated learning algorithms have been proposed for the non-convex distributed problem, federated learning in…

Federated Learning image-classification Image Classification Language Modeling+1

On the SDEs and Scaling Rules for Adaptive Gradient Algorithms

2022-05-20 · Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, Sanjeev Arora

Approximating Stochastic Gradient Descent (SGD) as a Stochastic Differential Equation (SDE) has allowed researchers to enjoy the benefits of studying a continuous optimization trajectory while carefully preserving the st…