AMSGrad

2000년 도입 · 논문 49편에서 사용

AMSGrad is a stochastic optimization method that seeks to fix a convergence issue with Adam based optimizers. AMSGrad uses the maximum of past squared gradients $v\_{t}$ rather than the exponential average to update the parameters: $$m\_{t} = \beta\_{1}m\_{t-1} + \left(1-\beta\_{1}\right)g\_{t} $$ $$v\_{t} = \beta\_{2}v\_{t-1} + \left(1-\beta\_{2}\right)g\_{t}^{2}$$ $$ \hat{v}\_{t} = \max\left(\hat{v}\_{t-1}, v\_{t}\right) $$ $$\theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{\hat{v}_{t}} + \epsilon}m\_{t}$$

출처: On the Convergence of Adam and Beyond

소개 논문: On the Convergence of Adam and Beyond