ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles

2023-06-12 · Kai Zhao, Jianye Hao, Yi Ma, Jinyi Liu, Yan Zheng, Zhaopeng Meng

Offline reinforcement learning (RL) is a learning paradigm where an agent learns from a fixed dataset of experience. However, learning solely from a static dataset can limit the performance due to the lack of exploration. To overcome it, offline-to-online RL combines offline pre-training with online fine-tuning, which enables the agent to further refine its policy by interacting with the environment in real-time. Despite its benefits, existing offline-to-online RL methods suffer from performance degradation and slow improvement during the online phase. To tackle these challenges, we propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Moreover, to expedite online performance enhancement, we appropriately loosen the pessimism of Q-value estimation and incorporate ensemble-based exploration mechanisms into our framework. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods during online fine-tuning on a range of locomotion and navigation tasks, significantly outperforming existing offline-to-online RL methods.

📄 PDF Abstract BibTeX arXiv:2306.06871

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

2024-02-06 · Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas B. Schön 외

This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distr…

D4RL Offline RL reinforcement-learning Reinforcement Learning (RL)

Offline Model-Based Reinforcement Learning with Anti-Exploration

2024-08-20 · Padmanaba Srinivasan, William Knottenbelt

Model-based reinforcement learning (MBRL) algorithms learn a dynamics model from collected data and apply it to generate synthetic trajectories to enable faster learning. This is an especially promising paradigm in offli…

D4RL model Model-based Reinforcement Learning Offline RL+3

Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning

2023-03-26 · Alex Beeson, Giovanni Montana

Offline reinforcement learning agents seek optimal policies from fixed data sets. With environmental interaction prohibited, agents face significant challenges in preventing errors in value estimates from compounding and…

Behavioural cloning Benchmarking reinforcement-learning

Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning

2024-02-05 · Abdelhakim Benechehab, Albert Thomas, Balázs Kégl

We consider the problem of offline reinforcement learning where only a set of system transitions is made available for policy optimization. Following recent advances in the field, we consider a model-based reinforcement …

D4RL Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning

Enhancing Robustness of Asynchronous EEG-Based Movement Prediction using Classifier Ensembles

2026-01-07 · Niklas Kueper, Kartik Chari, Elsa Andrea Kirchner arxiv

Objective: Stroke is one of the leading causes of disabilities. One promising approach is to extend the rehabilitation with self-initiated robot-assisted movement therapy. To enable this, it is required to detect the pat…