홈 › Papers

Offline Actor-Critic Reinforcement Learning Scales to Large Models

2024-02-08 · Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Heess, Martin Riedmiller

We show that offline actor-critic reinforcement learning can scale to large models - such as transformers - and follows similar scaling laws as supervised learning. We find that offline actor-critic algorithms can outperform strong, supervised, behavioral cloning baselines for multi-task training on a large dataset containing both sub-optimal and expert behavior on 132 continuous control tasks. We introduce a Perceiver-based actor-critic model and elucidate the key model features needed to make offline RL work with self- and cross-attention modules. Overall, we find that: i) simple offline actor critic algorithms are a natural choice for gradually moving away from the currently predominant paradigm of behavioral cloning, and ii) via offline RL it is possible to learn multi-task policies that master many domains simultaneously, including real robotics tasks, from sub-optimal demonstrations or self-generated data.

📄 PDF Abstract BibTeX arXiv:2402.05546

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control Offline RL reinforcement-learning Reinforcement Learning

KAN v.s. MLP for Offline Reinforcement Learning

2024-09-15 · Haihong Guo, Fengxin Li, Jiao Li, Hongyan Liu

Kolmogorov-Arnold Networks (KAN) is an emerging neural network architecture in machine learning. It has greatly interested the research community about whether KAN can be a promising alternative of the commonly used Mult…

D4RL Kolmogorov-Arnold Networks Offline RL Q-Learning+3

POPO: Pessimistic Offline Policy Optimization

2020-12-26 · Qiang He, Xinwen Hou

Offline reinforcement learning (RL), also known as batch RL, aims to optimize policy from a large pre-recorded dataset without interaction with the environment. This setting offers the promise of utilizing diverse, pre-c…

Offline RL Q-Learning Reinforcement Learning (RL)

Flow Actor-Critic for Offline Reinforcement Learning

2026-02-20 · Jongseong Chae, Jongeui Park, Yongjae Shin, Gyeongmin Kim 외 arxiv

The dataset distributions in offline reinforcement learning (RL) often exhibit complex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. …

Reinforcement Learning Offline RL

Simple Actors and Deep Critics for Scalable Reinforcement Learning

2026-08-27 · Guhyeon Kang, Jaehwi Lee, Minhae Kwon arxiv

Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these a…

Reinforcement Learning Offline RL

Adversarially Trained Actor Critic for Offline Reinforcement Learning

2022-02-05 · Ching-An Cheng, Tengyang Xie, Nan Jiang, Alekh Agarwal

We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism. ATAC is designed as a…

continuous-control Continuous Control D4RL Offline RL+3