OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

2024-05-27 · Allen Nie, Yash Chandak, Christina J. Yuan, Anirudhan Badrinath, Yannis Flet-Berliac, Emma Brunskil

Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.

📄 PDF Abstract BibTeX arXiv:2405.17708

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making Offline RLOff-policy evaluationSequential Decision Making

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Weighted model estimation for offline model-based reinforcement learning

2021-12-01 · NeurIPS 2021 12 · Toru Hishinuma, Kei Senda

This paper discusses model estimation in offline model-based reinforcement learning (MBRL), which is important for subsequent policy improvement using an estimated model. From the viewpoint of covariate shift, a natural …

Density Ratio Estimation model Model-based Reinforcement Learning Off-policy evaluation+3

Kernel weighted importance sampling for off-policy evaluation in contextual bandits

2026-07-16 · Joshua Spear, Matthieu Komorowski, Rebecca Pope, Neil J Sebire 외 arxiv

This article presents a novel estimator for performing off-policy evaluation using only offline data for contextual bandits. The proposed estimator, Kernel-WIS is demonstrated to be asymptotically consistent and to empir…

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

2023-01-30 · NeurIPS 2023 11 · Hanlin Zhu, Paria Rashidinejad, Jiantao Jiao

We propose A-Crab (Actor-Critic Regularized by Average Bellman error), a new practical algorithm for offline reinforcement learning (RL) in complex environments with insufficient data coverage. Our algorithm combines the…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure

2023-05-24 · Xumei Xi, Christina Lee Yu, Yudong Chen

We consider offline Reinforcement Learning (RL), where the agent does not interact with the environment and must rely on offline data collected using a behavior policy. Previous works provide policy evaluation guarantees…

Matrix Completion reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

FluxVLA Engine: A One-Stop VLA Engineering Platform for Embodied Intelligence

2026-09-15 · Yinhao Li, Weixin Mao, Zihan Lan, Jikun Rong 외 arxiv

Vision-language-action (VLA) models, world-action models (WAMs), and offline reinforcement learning methods are rapidly expanding the design space of embodied policies, yet turning these algorithms into reliable robot sy…

Reinforcement Learning