S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning

2021-03-10 · Samarth Sinha, Ajay Mandlekar, Animesh Garg

Offline reinforcement learning proposes to learn policies from large collected datasets without interacting with the physical environment. These algorithms have made it possible to learn useful skills from data that can then be deployed in the environment in real-world settings where interactions may be costly or dangerous, such as autonomous driving or factories. However, current algorithms overfit to the dataset they are trained on and exhibit poor out-of-distribution generalization to the environment when deployed. In this paper, we study the effectiveness of performing data augmentations on the state space, and study 7 different augmentation schemes and how they behave with existing offline RL algorithms. We then combine the best data performing augmentation scheme with a state-of-the-art Q-learning technique, and improve the function approximation of the Q-networks by smoothening out the learned state-action space. We experimentally show that using this Surprisingly Simple Self-Supervision technique in RL (S4RL), we significantly improve over the current state-of-the-art algorithms on offline robot learning environments such as MetaWorld [1] and RoboSuite [2,3], and benchmark datasets such as D4RL [4].

📄 PDF Abstract BibTeX arXiv:2103.06326

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous Driving D4RL Data Augmentation Offline RLOut-of-Distribution GeneralizationQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Simple Ingredients for Offline Reinforcement Learning

2024-03-19 · Edoardo Cetin, Andrea Tirinzoni, Matteo Pirotta, Alessandro Lazaric 외

Offline reinforcement learning algorithms have proven effective on datasets highly connected to the target downstream task. Yet, leveraging a novel testbed (MOOD) in which trajectories come from heterogeneous sources, we…

D4RL reinforcement-learning Reinforcement Learning

Simplifying Deep Reinforcement Learning via Self-Supervision

2021-06-10 · Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, Xia Hu

Supervised regression to demonstrations has been demonstrated to be a stable way to train deep policy networks. We are motivated to study how we can take full advantage of supervised loss functions for stably training de…

Deep Reinforcement Learning regression reinforcement-learning Reinforcement Learning+1

Behavior Regularized Offline Reinforcement Learning

2019-11-26 · Yifan Wu, George Tucker, Ofir Nachum

In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline …

continuous-control Continuous Control Offline RL reinforcement-learning+2

Pre-training with Synthetic Data Helps Offline Reinforcement Learning

2023-10-01 · Zecheng Wang, Che Wang, Zixuan Dong, Keith Ross

Recently, it has been shown that for offline deep reinforcement learning (DRL), pre-training Decision Transformer with a large language corpus can improve downstream performance (Reid et al., 2022). A natural question to…

D4RL Deep Reinforcement Learning Q-Learning reinforcement-learning+1

Order-Optimal Instance-Dependent Bounds for Offline Reinforcement Learning with Preference Feedback

2024-06-18 · Zhirui Chen, Vincent Y. F. Tan

We consider offline reinforcement learning (RL) with preference feedback in which the implicit reward is a linear function of an unknown parameter. Given an offline dataset, our objective consists in ascertaining the opt…

Offline RL Reinforcement Learning (RL)