Multi-step Reinforcement Learning: A Unifying Algorithm

2017-03-03 · Kristopher De Asis, J. Fernando Hernandez-Garcia, G. Zacharias Holland, Richard S. Sutton

Unifying seemingly disparate algorithmic ideas to produce better performing algorithms has been a longstanding goal in reinforcement learning. As a primary example, TD($\lambda$) elegantly unifies one-step TD prediction with Monte Carlo methods through the use of eligibility traces and the trace-decay parameter $\lambda$. Currently, there are a multitude of algorithms that can be used to perform TD control, including Sarsa, $Q$-learning, and Expected Sarsa. These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance. Each of these algorithms is seemingly distinct, and no one dominates the others for all problems. In this paper, we study a new multi-step action-value algorithm called $Q(\sigma)$ which unifies and generalizes these existing algorithms, while subsuming them as special cases. A new parameter, $\sigma$, is introduced to allow the degree of sampling performed by the algorithm at each step during its backup to be continuously varied, with Sarsa existing at one extreme (full sampling), and Expected Sarsa existing at the other (pure expectation). $Q(\sigma)$ is generally applicable to both on- and off-policy learning, but in this work we focus on experiments in the on-policy case. Our results show that an intermediate value of $\sigma$, which results in a mixture of the existing algorithms, performs better than either extreme. The mixture can also be varied dynamically which can result in even greater performance.

📄 PDF Abstract BibTeX arXiv:1703.01327

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Expected Sarsa Expected Sarsa is like Q-learning but instead of taking the maximum over next state-action pairs, we use the expected value,…

Sarsa Sarsa is an on-policy TD control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} + \gamma{Q}\left(S\_{t+1},…

Double Q($σ$) and Q($σ, λ$): Unifying Reinforcement Learning Control Algorithms

2017-11-05 · Markus Dumke

Temporal-difference (TD) learning is an important field in reinforcement learning. Sarsa and Q-Learning are among the most used TD algorithms. The Q($\sigma$) algorithm (Sutton and Barto (2017)) unifies both. This paper …

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Unifying Framework for Reinforcement Learning and Planning

2020-06-26 · Thomas M. Moerland, Joost Broekens, Aske Plaat, Catholijn M. Jonker

Sequential decision making, commonly formalized as optimization of a Markov Decision Process, is a key challenge in artificial intelligence. Two successful approaches to MDP optimization are reinforcement learning and pl…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Probability Functional Descent: A Unifying Perspective on GANs, Variational Inference, and Reinforcement Learning

2019-01-30 · Casey Chu, Jose Blanchet, Peter Glynn

This paper provides a unifying view of a wide range of problems of interest in machine learning by framing them as the minimization of functionals defined on the space of probability measures. In particular, we show that…

BIG-bench Machine Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning

2024-02-05 · Peter Vamplew, Cameron Foale, Conor F. Hayes, Patrick Mannion 외

Research in multi-objective reinforcement learning (MORL) has introduced the utility-based paradigm, which makes use of both environmental rewards and a function that defines the utility derived by the user from those re…

Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Tutorial Introduction to Reinforcement Learning

2023-04-03 · Mathukumalli Vidyasagar

In this paper, we present a brief survey of Reinforcement Learning (RL), with particular emphasis on Stochastic Approximation (SA) as a unifying theme. The scope of the paper includes Markov Reward Processes, Markov Deci…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1