Demon CM

2000년 도입 · 논문 1편에서 사용

Demon CM, or SGD with Momentum and Demon, is the Demon momentum rule applied to SGD with momentum. $$ \beta\_{t} = \beta\_{init}\cdot\frac{\left(1-\frac{t}{T}\right)}{\left(1-\beta\_{init}\right) + \beta\_{init}\left(1-\frac{t}{T}\right)} $$ $$ \theta\_{t+1} = \theta\_{t} - \eta{g}\_{t} + \beta\_{t}v\_{t} $$ $$ v\_{t+1} = \beta\_{t}{v\_{t}} - \eta{g\_{t}} $$

출처: Demon: Improved Neural Network Training with Momentum Decay

소개 논문: Demon: Improved Neural Network Training with Momentum Decay