Selective Uncertainty Propagation in Offline RL
We consider the finite-horizon offline reinforcement learning (RL) setting, and are motivated by the challenge of learning the policy at any step h in dynamic programming (DP) algorithms. To learn this, it is sufficient to evaluate the treatment effect of deviating from the behavioral policy at step h after having optimized the policy for all future steps. Since the policy at any step can affect next-state distributions, the related distributional shift challenges can make this problem far more statistically hard than estimating such treatment effects in the stochastic contextual bandit setting. However, the hardness of many real-world RL instances lies between the two regimes. We develop a flexible and general method called selective uncertainty propagation for confidence interval construction that adapts to the hardness of the associated distribution shift challenges. We show benefits of our approach on toy environments and demonstrate the benefits of these techniques for offline policy learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Offline RLreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Selective Mask Propagation for Multi-Object Tracking
In multi-object tracking, most frames are easy for a lightweight base tracker while a small fraction is intrinsically hard. Video object segmentation (VOS) models can often preserve identity through the hard frames where…
Video Object SegmentationMulti-Object TrackingBayesian Uncertainty Propagation for Agentic RAG Pipelines: A Proof-of-Concept Study on Multi-Hop Question Answering
Trustworthy deployment of Agentic Retrieval-Augmented Generation (RAG) systems requires mechanisms for estimating when multi-stage reasoning pipelines may fail. This paper presents an uncertainty-aware Agentic Retrieval-…
Multi-hop Question AnsweringNonlinearity and Uncertainty Informed Moment-Matching Gaussian Mixture Splitting
Many problems in navigation and tracking require increasingly accurate characterizations of the evolution of uncertainty in nonlinear systems. Nonlinear uncertainty propagation approaches based on Gaussian mixture densit…
Computational EfficiencyDeterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning
Current approaches to model-based offline reinforcement learning often incorporate uncertainty-based reward penalization to address the distributional shift problem. These approaches, commonly known as pessimistic value …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Variational InferencePuzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning
Offline multi-agent reinforcement learning (MARL) aims to solve cooperative decision-making problems in multi-agent systems using pre-collected datasets. Existing offline MARL methods primarily constrain training within …
Multi-agent Reinforcement Learning