Tuning Learning Rates with the Cumulative-Learning Constant
This paper introduces a novel method for optimizing learning rates in machine learning. A previously unrecognized proportionality between learning rates and dataset sizes is discovered, providing valuable insights into how dataset scale influences training dynamics. Additionally, a cumulative learning constant is identified, offering a framework for designing and optimizing advanced learning rate schedules. These findings have the potential to enhance training efficiency and performance across a wide range of machine learning applications.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Agile Online Model Selection: Resolving Adaptation Lag via Safeguarded Large Learning Rates
Maintaining predictive accuracy in non-stationary environments requires online model selection to adapt autonomously to unknown distribution shifts. However, existing tuning-free algorithms face a fundamental trade-off b…
From Non-Convex to Strongly Convex: Curvature-Adaptive FTPL for Online Optimization
Curvature adaptivity is a classical theme in online optimization: for convex Lipschitz losses, adaptive methods interpolate between the optimal $O(\sqrt{T})$ regret for general convex losses and $O(\log T)$ regret under …
Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates
We provide a new understanding of the stochastic gradient bandit algorithm by showing that it converges to a globally optimal policy almost surely using \emph{any} constant learning rate. This result demonstrates that th…
A Dynamic Observation Strategy for Multi-agent Multi-armed Bandit Problem
We define and analyze a multi-agent multi-armed bandit problem in which decision-making agents can observe the choices and rewards of their neighbors under a linear observation cost. Neighbors are defined by a network gr…
Decision MakingNear-Constant Strong Violation and Last-Iterate Convergence for Online CMDPs via Decaying Safety Margins
We study safe online reinforcement learning in Constrained Markov Decision Processes (CMDPs) under strong regret and violation metrics, which forbid error cancellation over time. Existing primal-dual methods that achieve…
Reinforcement Learning