Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters

2021-10-08 · Vladislav Kurenkov, Sergey Kolesnikov

In this work, we argue for the importance of an online evaluation budget for a reliable comparison of deep offline RL algorithms. First, we delineate that the online evaluation budget is problem-dependent, where some problems allow for less but others for more. And second, we demonstrate that the preference between algorithms is budget-dependent across a diverse range of decision-making domains such as Robotics, Finance, and Energy Management. Following the points above, we suggest reporting the performance of deep offline RL algorithms under varying online evaluation budgets. To facilitate this, we propose to use a reporting tool from the NLP field, Expected Validation Performance. This technique makes it possible to reliably estimate expected maximum performance under different budgets while not requiring any additional computation beyond hyperparameter search. By employing this tool, we also show that Behavioral Cloning is often more favorable to offline RL algorithms when working within a limited budget.

📄 PDF Abstract BibTeX arXiv:2110.04156

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Makingenergy managementManagement Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning

2022-06-27 · Haoyi Niu, Shubham Sharma, Yiwen Qiu, Ming Li 외

Learning effective reinforcement learning (RL) policies to solve real-world complex tasks can be quite challenging without a high-fidelity simulation environment. In most cases, we are only given imperfect simulators wit…

Offline RL reinforcement-learning Reinforcement Learning (RL)

Active Advantage-Aligned Online Reinforcement Learning with Offline Data

2025-02-11 · Xuefeng Liu, Hung T. C. Le, Siyu Chen, Rick Stevens 외

Online reinforcement learning (RL) enhances policies through direct interactions with the environment, but faces challenges related to sample efficiency. In contrast, offline RL leverages extensive pre-collected data to …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

MOORL: A Framework for Integrating Offline-Online Reinforcement Learning

2025-06-11 · Gaurav Chaudhary, Wassim Uddin Mondal, Laxmidhar Behera

Sample efficiency and exploration remain critical challenges in Deep Reinforcement Learning (DRL), particularly in complex domains. Offline RL, which enables agents to learn optimal policies from static, pre-collected da…

D4RL Deep Reinforcement Learning Efficient Exploration Offline RL+2

Spectral Policy Optimization: Coloring your Incorrect Reasoning in GRPO

2025-05-16 · Peter Chen, Xiaopeng Li, Ziniu Li, Xi Chen 외

Reinforcement learning (RL) has demonstrated significant success in enhancing reasoning capabilities in large language models (LLMs). One of the most widely used RL methods is Group Relative Policy Optimization (GRPO)~\c…

All Diversity Reinforcement Learning (RL)

From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning

2025-11-05 · Lipeng Zu, Yu Qian, Shayok Chakraborty, Xiaonan Zhang arxiv

Offline-to-online reinforcement learning (O2O RL) faces a central challenge between retaining offline conservatism and adapting to online feedback under distribution shift. This challenge arises because data behavior evo…

Reinforcement Learning