Corruption-Robust Offline Reinforcement Learning

2021-06-11 · Xuezhou Zhang, Yiding Chen, Jerry Zhu, Wen Sun

We study the adversarial robustness in offline reinforcement learning. Given a batch dataset consisting of tuples $(s, a, r, s')$, an adversary is allowed to arbitrarily modify $\epsilon$ fraction of the tuples. From the corrupted dataset the learner aims to robustly identify a near-optimal policy. We first show that a worst-case $\Omega(d\epsilon)$ optimality gap is unavoidable in linear MDP of dimension $d$, even if the adversary only corrupts the reward element in a tuple. This contrasts with dimension-free results in robust supervised learning and best-known lower-bound in the online RL setting with corruption. Next, we propose robust variants of the Least-Square Value Iteration (LSVI) algorithm utilizing robust supervised learning oracles, which achieve near-matching performances in cases both with and without full data coverage. The algorithm requires the knowledge of $\epsilon$ to design the pessimism bonus in the no-coverage case. Surprisingly, in this case, the knowledge of $\epsilon$ is necessary, as we show that being adaptive to unknown $\epsilon$ is impossible.This again contrasts with recent results on corruption-robust online RL and implies that robust offline RL is a strictly harder problem.

📄 PDF Abstract BibTeX arXiv:2106.06630

Code (0)

등록된 구현이 없습니다.

Tasks

Adversarial Robustness Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Corruption Robust Offline Reinforcement Learning with Human Feedback

2024-02-09 · Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban, Adish Singla 외

We study data corruption robustness for reinforcement learning with human feedback (RLHF) in an offline setting. Given an offline dataset of pairs of trajectories along with feedback about human preferences, an $\varepsi…

Adversarial Attack reinforcement-learning Reinforcement Learning

Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization

2025-11-14 · Le Xu, Jiayu Chen arxiv

Offline reinforcement learning (RL) is vulnerable to real-world data corruption, with even robust algorithms failing under challenging observation and mixture corruptions. We posit this failure stems from data corruption…

Reinforcement Learning Offline RL

Corruption-Robust Offline Reinforcement Learning with General Function Approximation

2023-10-23 · NeurIPS 2023 11 · Chenlu Ye, Rui Yang, Quanquan Gu, Tong Zhang

We investigate the problem of corruption robustness in offline reinforcement learning (RL) with general function approximation, where an adversary can corrupt each sample in the offline dataset, and the corruption level …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

2023-10-19 · Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang 외

Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collecte…

Offline RL Q-Learning reinforcement-learning Reinforcement Learning+1

Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions

2024-11-01 · Rui Yang, Jie Wang, Guoping Wu, Bin Li

Real-world offline datasets are often subject to data corruptions (such as noise or adversarial attacks) due to sensor failures or malicious attacks. Despite advances in robust offline reinforcement learning (RL), existi…

Bayesian Inference Offline RL Reinforcement Learning (RL)