Polyak Averaging

1991년 도입 · 논문 3편에서 사용

Polyak Averaging is an optimization technique that sets final parameters to an average of (recent) parameters visited in the optimization trajectory. Specifically if in $t$ iterations we have parameters $\theta\_{1}, \theta\_{2}, \dots, \theta\_{t}$, then Polyak Averaging suggests setting $$ \theta\_t =\frac{1}{t}\sum\_{i}\theta\_{i} $$ Image Credit: Shubhendu Trivedi & Risi Kondor