Demon ADAM

2000년 도입 · 논문 1편에서 사용

Demon Adam is a stochastic optimizer where the Demon momentum rule is applied to the Adam optimizer. $$ \beta\_{t} = \beta\_{init}\cdot\frac{\left(1-\frac{t}{T}\right)}{\left(1-\beta\_{init}\right) + \beta\_{init}\left(1-\frac{t}{T}\right)} $$ $$ m\_{t, i} = g\_{t, i} + \beta\_{t}m\_{t-1, i} $$ $$ v\_{t+1} = \beta\_{2}v\_{t} + \left(1-\beta\_{2}\right)g^{2}\_{t} $$ $$ \theta_{t} = \theta_{t-1} - \eta\frac{\hat{m}\_{t}}{\sqrt{\hat{v}\_{t}} + \epsilon} $$

출처: Demon: Improved Neural Network Training with Momentum Decay

소개 논문: Demon: Improved Neural Network Training with Momentum Decay