Practical Open-Loop Optimistic Planning
We consider the problem of online planning in a Markov Decision Process when given only access to a generative model, restricted to open-loop policies - i.e. sequences of actions - and under budget constraint. In this setting, the Open-Loop Optimistic Planning (OLOP) algorithm enjoys good theoretical guarantees but is overly conservative in practice, as we show in numerical experiments. We propose a modified version of the algorithm with tighter upper-confidence bounds, KLOLOP, that leads to better practical performances while retaining the sample complexity bound. Finally, we propose an efficient implementation that significantly improves the time complexity of both algorithms.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Open-loop POMDP Simplification and Safe Skipping of Replanning with Formal Performance Guarantees
Partially Observable Markov Decision Processes (POMDPs) provide a principled mathematical framework for decision-making under uncertainty. However, the exact solution to POMDPs is computationally intractable. In this pap…
Adaptive Thompson Sampling Stacks for Memory Bounded Open-Loop Planning
We propose Stable Yet Memory Bounded Open-Loop (SYMBOL) planning, a general memory bounded approach to partially observable open-loop planning. SYMBOL maintains an adaptive stack of Thompson Sampling bandits, whose size …
Thompson SamplingHuman-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation
We study human-in-the-loop reinforcement learning (RL) with trajectory preferences, where instead of receiving a numeric reward at each step, the agent only receives preferences over trajectory pairs from a human oversee…
Reinforcement Learning (RL)Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
Model-based reinforcement learning algorithms with probabilistic dynamical models are amongst the most data-efficient learning methods. This is often attributed to their ability to distinguish between epistemic and aleat…
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Open-Loop Planning, Closed-Loop Verification: Speculative Verification for VLA
Vision-Language-Action (VLA) models, as large foundation models for embodied control, have shown strong performance in manipulation tasks. However, their performance comes at high inference cost. To improve efficiency, r…