Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

2024-05-28 · Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, Xianyuan Zhan

Off-policy reinforcement learning (RL) has achieved notable success in tackling many complex real-world tasks, by leveraging previously collected data for policy learning. However, most existing off-policy RL algorithms fail to maximally exploit the information in the replay buffer, limiting sample efficiency and policy performance. In this work, we discover that concurrently training an offline RL policy based on the shared online replay buffer can sometimes outperform the original online learning policy, though the occurrence of such performance gains remains uncertain. This motivates a new possibility of harnessing the emergent outperforming offline optimal policy to improve online policy learning. Based on this insight, we present Offline-Boosted Actor-Critic (OBAC), a model-free online RL framework that elegantly identifies the outperforming offline policy through value comparison, and uses it as an adaptive constraint to guarantee stronger policy learning performance. Our experiments demonstrate that OBAC outperforms other popular model-free RL baselines and rivals advanced model-based RL methods in terms of sample efficiency and asymptotic performance across 53 tasks spanning 6 task suites.

📄 PDF Abstract BibTeX arXiv:2405.18520

Code (1)

roythuly/obac 공식 구현 pytorch

Tasks

Offline RL Reinforcement Learning (RL)

Improving Offline RL by Blending Heuristics

2023-06-01 · Sinong Geng, Aldo Pacchiano, Andrey Kolobov, Ching-An Cheng

We propose Heuristic Blending (HUBL), a simple performance-improving technique for a broad class of offline RL algorithms based on value bootstrapping. HUBL modifies the Bellman operators used in these algorithms, partia…

D4RL Offline RL

ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network

2026-05-10 · Qian Chen, Junqiao Zhao, Hongtu Zhou, Hang Yu 외 arxiv

Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic metho…

Reinforcement Learning Offline RL

Offline Actor-Critic Reinforcement Learning Scales to Large Models

2024-02-08 · Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth 외

We show that offline actor-critic reinforcement learning can scale to large models - such as transformers - and follows similar scaling laws as supervised learning. We find that offline actor-critic algorithms can outper…

continuous-control Continuous Control Offline RL reinforcement-learning+1

SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

2026-02-19 · Nathan Samuel de Lara, Florian Shkurti arxiv

Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance. We provide…

Reinforcement Learning Offline RL

Blending gradient boosted trees and neural networks for point and probabilistic forecasting of hierarchical time series

2023-10-19 · International Journal of Forecasting 2022 2 · Ioannis Nasios, Konstantinos Vogklis

In this paper we tackle the problem of point and probabilistic forecasting by describing a blending methodology of machine learning models that belong to gradient boosted trees and neural networks families. These princip…

Diversity Feature Engineering Time Series