홈 › Papers

Uncertainty Weighted Offline Reinforcement Learning

2021-01-01 · Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, Hanlin Goh

Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We hypothesize that a key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that models the epistemic uncertainty to detect OOD state-action pairs and down-weights their contribution in the training objectives accordingly. Implementation-wise, we adopt a practical and effective dropout-based uncertainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-art baseline on datasets with sparse demonstrations collected from human experts.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RLQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning

2021-05-17 · Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind 외

Offline RL Q-Learning reinforcement-learning Reinforcement Learning+1

Fine-tuning Behavioral Cloning Policies with Preference-Based Reinforcement Learning

2025-09-30 · Maël Macuglia, Paul Friedrich, Giorgia Ramponi arxiv

Deploying reinforcement learning (RL) in robotics, industry, and health care is blocked by two obstacles: the difficulty of specifying accurate rewards and the risk of unsafe, data-hungry exploration. We address this by …

Reinforcement Learning Continuous Control

A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

2023-06-13 · Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang 외

Offline reinforcement learning (RL) provides a promising solution to learning an agent fully relying on a data-driven paradigm. However, constrained by the limited quality of the offline dataset, its performance is often…

D4RL Efficient Exploration Offline RL reinforcement-learning+1

Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning

2023-07-01 · PMLR 2023 7 · Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin 외

For offline reinforcement learning (RL), model-based methods are expected to be data-efficient as they incorporate dynamics models to generate more data. However, due to inevitable model errors, straightforwardly learnin…

D4RL model Offline RL reinforcement-learning+3

Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning

2022-06-02 · David Brandfonbrener, Remi Tachet des Combes, Romain Laroche

Most theoretically motivated work in the offline reinforcement learning setting requires precise uncertainty estimates. This requirement restricts the algorithms derived in that work to the tabular and linear settings wh…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)