Cosine Annealing

2000년 도입 · 논문 3,965편에서 사용

Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before being increased rapidly again. The resetting of the learning rate acts like a simulated restart of the learning process and the re-use of good weights as the starting point of the restart is referred to as a "warm restart" in contrast to a "cold restart" where a new set of small random numbers may be used as a starting point. $$\eta\_{t} = \eta\_{min}^{i} + \frac{1}{2}\left(\eta\_{max}^{i}-\eta\_{min}^{i}\right)\left(1+\cos\left(\frac{T\_{cur}}{T\_{i}}\pi\right)\right) $$ Where where $\eta\_{min}^{i}$ and $ \eta\_{max}^{i}$ are ranges for the learning rate, and $T\_{cur}$ account for how many epochs have been performed since the last restart. Text Source: Jason Brownlee Image Source: Gao Huang

출처: SGDR: Stochastic Gradient Descent with Warm Restarts

소개 논문: SGDR: Stochastic Gradient Descent with Warm Restarts