TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning

2024-10-12 · Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, Gerhard Neumann

This work introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), a novel algorithm that enables off-policy updates in the ERL framework. In ERL, policies predict entire action trajectories over multiple time steps instead of single actions at every time step. These trajectories are typically parameterized by trajectory generators such as Movement Primitives (MP), allowing for smooth and efficient exploration over long horizons while capturing high-level temporal correlations. However, ERL methods are often constrained to on-policy frameworks due to the difficulty of evaluating state-action values for entire action sequences, limiting their sample efficiency and preventing the use of more efficient off-policy architectures. TOP-ERL addresses this shortcoming by segmenting long action sequences and estimating the state-action values for each segment using a transformer-based critic architecture alongside an n-step return estimation. These contributions result in efficient and stable training that is reflected in the empirical results conducted on sophisticated robot learning environments. TOP-ERL significantly outperforms state-of-the-art RL methods. Thorough ablation studies additionally show the impact of key design choices on the model performance.

📄 PDF Abstract BibTeX arXiv:2410.09536

Code (1)

brucegeli/top_erl_iclr25 공식 구현

Tasks

Efficient Explorationreinforcement-learning Reinforcement Learning

Episodic Policy Gradient Training

2021-12-03 · Hung Le, Majid Abdolshah, Thommen K. George, Kien Do 외

We introduce a novel training procedure for policy gradient methods wherein episodic memory is used to optimize the hyperparameters of reinforcement learning algorithms on-the-fly. Unlike other hyperparameter searches, w…

Policy Gradient Methods Scheduling

Two-Memory Reinforcement Learning

2023-04-20 · Zhao Yang, Thomas. M. Moerland, Mike Preuss, Aske Plaat

While deep reinforcement learning has shown important empirical success, it tends to learn relatively slow due to slow propagation of rewards information and slow update of parametric neural networks. Non-parametric epis…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Representation Learning+1

Transformers are Meta-Reinforcement Learners

2021-09-29 · Luckeciano Carvalho Melo

The transformer architecture and variants presented a remarkable success across many machine learning tasks in recent years. This success is intrinsically related to the capability of handling long sequences and the pres…

continuous-control Continuous Control Meta Reinforcement Learning Out-of-Distribution Generalization+3

e-COP : Episodic Constrained Optimization of Policies

2024-06-13 · Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Sahil Singla

In this paper, we present the $\texttt{e-COP}$ algorithm, the first policy optimization algorithm for constrained Reinforcement Learning (RL) in episodic (finite horizon) settings. Such formulations are applicable when t…

LEMMA reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methods

2021-09-13 · Xin Guo, Anran Hu, Junzi Zhang

When designing algorithms for finite-time-horizon episodic reinforcement learning problems, a common approach is to introduce a fictitious discount factor and use stationary policies for approximations. Empirically, it h…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)