Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data

2022-10-16 · Allen Nie, Yannis Flet-Berliac, Deon R. Jordan, William Steenbergen, Emma Brunskill

Offline reinforcement learning (RL) can be used to improve future performance by leveraging historical data. There exist many different algorithms for offline RL, and it is well recognized that these algorithms, and their hyperparameter settings, can lead to decision policies with substantially differing performance. This prompts the need for pipelines that allow practitioners to systematically perform algorithm-hyperparameter selection for their setting. Critically, in most real-world settings, this pipeline must only involve the use of historical data. Inspired by statistical model selection methods for supervised learning, we introduce a task- and method-agnostic pipeline for automatically training, comparing, selecting, and deploying the best policy when the provided dataset is limited in size. In particular, our work highlights the importance of performing multiple data splits to produce more reliable algorithm-hyperparameter selection. While this is a common approach in supervised learning, to our knowledge, this has not been discussed in detail in the offline RL setting. We show it can have substantial impacts when the dataset is small. Compared to alternate approaches, our proposed pipeline outputs higher-performing deployed policies from a broad range of offline policy learning algorithms and across various simulation domains in healthcare, education, and robotics. This work contributes toward the development of a general-purpose meta-algorithm for automatic algorithm-hyperparameter selection for offline RL.

📄 PDF Abstract BibTeX arXiv:2210.08642

Code (0)

등록된 구현이 없습니다.

Tasks

Model SelectionOffline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching

2024-02-04 · Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long 외

In offline reinforcement learning (RL), the performance of the learned policy highly depends on the quality of offline datasets. However, in many cases, the offline dataset contains very limited optimal trajectories, whi…

D4RL Data Augmentation Imitation Learning Offline RL+2

ARLO: A Framework for Automated Reinforcement Learning

2022-05-20 · Marco Mussi, Davide Lombarda, Alberto Maria Metelli, Francesco Trovò 외

Automated Reinforcement Learning (AutoRL) is a relatively new area of research that is gaining increasing attention. The objective of AutoRL consists in easing the employment of Reinforcement Learning (RL) techniques for…

feature selection MuJoCo reinforcement-learning Reinforcement Learning+1

Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

2026-05-06 · Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang, Miroslav Pajic 외 arxiv

In offline-to-online reinforcement learning (O2O-RL), policies are first safely trained offline using previously collected datasets and then further fine-tuned for tasks via limited online interactions. In a typical O2O-…

Reinforcement Learning Offline RL

Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity

2025-06-20 · Samin Yeasar Arnob, Scott Fujimoto, Doina Precup

In this paper, we investigate the use of small datasets in the context of offline reinforcement learning (RL). While many common offline RL benchmarks employ datasets with over a million data points, many offline RL appl…

continuous-control Continuous Control Offline RL reinforcement-learning+2

Shortcut Trajectory Planning for Efficient Offline Reinforcement Learning

2026-07-10 · Guanquan Wang, Yoshimasa Tsuruoka arxiv

Diffusion-based trajectory planners have shown strong performance in offline reinforcement learning, but their iterative denoising process often incurs high inference cost. Consistency-based planners reduce the number of…

Reinforcement Learning Trajectory Planning