Offline Reinforcement Learning at Multiple Frequencies

2022-07-26 · Kaylee Burns, Tianhe Yu, Chelsea Finn, Karol Hausman

Leveraging many sources of offline robot data requires grappling with the heterogeneity of such data. In this paper, we focus on one particular aspect of heterogeneity: learning from offline data collected at different control frequencies. Across labs, the discretization of controllers, sampling rates of sensors, and demands of a task of interest may differ, giving rise to a mixture of frequencies in an aggregated dataset. We study how well offline reinforcement learning (RL) algorithms can accommodate data with a mixture of frequencies during training. We observe that the $Q$-value propagates at different rates for different discretizations, leading to a number of learning challenges for off-the-shelf offline RL. We present a simple yet effective solution that enforces consistency in the rate of $Q$-value updates to stabilize learning. By scaling the value of $N$ in $N$-step returns with the discretization size, we effectively balance $Q$-value propagation, leading to more stable convergence. On three simulated robotic control problems, we empirically find that this simple approach outperforms na\"ive mixing by 50% on average.

📄 PDF Abstract BibTeX arXiv:2207.13082

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples

2023-03-07 · Mustafa O. Karabag, Ufuk Topcu

Offline reinforcement learning (offline RL) considers problems where learning is performed using only previously collected samples and is helpful for the settings in which collecting new data is costly or risky. In model…

Offline RL Off-policy evaluation reinforcement-learning Reinforcement Learning+1

Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning

2024-06-10 · Takayuki Osa, Tatsuya Harada

Recent studies on online reinforcement learning (RL) have demonstrated the advantages of learning multiple behaviors from a single task, as in the case of few-shot adaptation to a new environment. Although this approach …

Offline RL Reinforcement Learning (RL)

Reinforcement Learning for Control with Multiple Frequencies

2020-12-01 · NeurIPS 2020 12 · Jongmin Lee, ByungJun Lee, Kee-Eung Kim

Many real-world sequential decision problems involve multiple action variables whose control frequencies are different, such that actions take their effects at different periods. While these problems can be formulated wi…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Conservative Data Sharing for Multi-Task Offline Reinforcement Learning

2021-09-16 · NeurIPS 2021 12 · Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman 외

Offline reinforcement learning (RL) algorithms have shown promising results in domains where abundant pre-collected data is available. However, prior methods focus on solving individual problems from scratch with an offl…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning

2026-05-29 · Ha Manh Bui, Metod Jazbec, Eric Nalisnick, Anqi Liu arxiv

Offline-to-Online Reinforcement Learning (O2O-RL) leverages an offline, pre-trained policy to minimize costly online interactions. Although data-efficient, O2O-RL is susceptible to shifts between offline and online distr…

Reinforcement Learning