Cyclical Log Annealing as a Learning Rate Scheduler

2024-03-13 · Philip Naveen

A learning rate scheduler is a predefined set of instructions for varying search stepsizes during model training processes. This paper introduces a new logarithmic method using harsh restarting of step sizes through stochastic gradient descent. Cyclical log annealing implements the restart pattern more aggressively to maybe allow the usage of more greedy algorithms on the online convex optimization framework. The algorithm was tested on the CIFAR-10 image datasets, and seemed to perform analogously with cosine annealing on large transformer-enhanced residual neural networks. Future experiments would involve testing the scheduler in generative adversarial networks and finding the best parameters for the scheduler with more experiments.

📄 PDF Abstract BibTeX arXiv:2403.14685

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

When to restart? Exploring escalating restarts on convergence

2026-03-04 · Ayush K. Varshney, Šarūnas Girdzijauskas, Konstantinos Vandikas, Aneta Vulgarakis Feljan arxiv

Learning rate scheduling plays a critical role in the optimization of deep neural networks, directly influencing convergence speed, stability, and generalization. While existing schedulers such as cosine annealing, cycli…

Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing

2019-03-25 · NAACL 2019 6 · Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao 외

Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing (NLP) tasks. The VAE objective consists of two terms, (i) reconstruction and (ii) KL regularization, …

Decoder Language Modeling Language Modelling Response Generation+1

Stochastic Gradient Descent with Hyperbolic-Tangent Decay on Classification

2018-06-05 · Bo Yang Hsueh, Wei Li, I-Chen Wu

Learning rate scheduler has been a critical issue in the deep neural network training. Several schedulers and methods have been proposed, including step decay scheduler, adaptive method, cosine scheduler and cyclical sch…

Classification General Classification Scheduling

AdaAnn: Adaptive Annealing Scheduler for Probability Density Approximation

2022-02-01 · Emma R. Cobian, Jonathan D. Hauenstein, Fang Liu, Daniele E. Schiavazzi

Approximating probability distributions can be a challenging task, particularly when they are supported over regions of high geometrical complexity or exhibit multiple modes. Annealing can be used to facilitate this task…

Computational Efficiency parameter estimation Variational Inference

SALSA: Simulated Annealing based Loop-Ordering Scheduler for DNN Accelerators

2023-04-20 · Victor J. B. Jung, Arne Symons, Linyan Mei, Marian Verhelst 외

To meet the growing need for computational power for DNNs, multiple specialized hardware architectures have been proposed. Each DNN layer should be mapped onto the hardware with the most efficient schedule, however, SotA…