N-step Returns

2000년 도입 · 논문 29편에서 사용

$n$-step Returns are used for value function estimation in reinforcement learning. Specifically, for $n$ steps we can write the complete return as: $$ R\_{t}^{(n)} = r\_{t+1} + \gamma{r}\_{t+2} + \cdots + \gamma^{n-1}\_{t+n} + \gamma^{n}V\_{t}\left(s\_{t+n}\right) $$ We can then write an $n$-step backup, in the style of TD learning, as: $$ \Delta{V}\_{r}\left(s\_{t}\right) = \alpha\left[R\_{t}^{(n)} - V\_{t}\left(s\_{t}\right)\right] $$ Multi-step returns often lead to faster learning with suitably tuned $n$. Image Credit: Sutton and Barto, Reinforcement Learning