Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators
In many reinforcement learning (RL) applications one cannot easily let the agent act in the world; this is true for autonomous vehicles, healthcare applications, and even some recommender systems, to name a few examples. Offline RL provides a way to train agents without real-world exploration, but is often faced with biases due to data distribution shifts, limited coverage, and incomplete representation of the environment. To address these issues, practical applications have tried to combine simulators with grounded offline data, using so-called hybrid methods. However, constructing a reliable simulator is in itself often challenging due to intricate system complexities as well as missing or incomplete information. In this work, we outline four principal challenges for combining offline data with imperfect simulators in RL: simulator modeling error, partial observability, state and action discrepancies, and hidden confounding. To help drive the RL community to pursue these problems, we construct ``Benchmarks for Mechanistic Offline Reinforcement Learning'' (B4MRL), which provide dataset-simulator benchmarks for the aforementioned challenges. Our results suggest the key necessity of such benchmarks for future research.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous VehiclesOffline RLRecommendation Systemsreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Survival Instinct in Offline Reinforcement Learning
We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with "wrong" rew…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Mind the Gap: Offline Policy Optimization for Imperfect Rewards
Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously difficult to design. In many cases, only imperfect rewar…
Reinforcement Learning (RL)Data-Augmented Game Starts for Accelerating Self-Play Exploration in Imperfect Information Games
Finding approximate equilibria for large-scale imperfect-information competitive games such as StarCraft, Dota, and CounterStrike remains computationally infeasible due to sparse rewards and challenging exploration over …
Reinforcement LearningOASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning
Offline safe reinforcement learning (RL) aims to train a policy that satisfies constraints using a pre-collected dataset. Most current methods struggle with the mismatch between imperfect demonstrations and the desired s…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningCounterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation
In applying reinforcement learning (RL) to high-stakes domains, quantitative and qualitative evaluation using observational data can help practitioners understand the generalization performance of new policies. However, …
counterfactualOff-policy evaluationReinforcement Learning (RL)