Relating Reinforcement Learning to Dynamic Programming-Based Planning

2026-03-08 · Filip V. Georgiev, Kalle G. Timperi, Başak Sakçak, Steven M. LaValle arxiv

This paper bridges some of the gap between optimal planning and reinforcement learning (RL), both of which share roots in dynamic programming applied to sequential decision making or optimal control. Whereas planning typically favors deterministic models, goal termination, and cost minimization, RL tends to favor stochastic models, infinite-horizon discounting, and reward maximization in addition to learning-related parameters such as the learning rate and greediness factor. A derandomized version of RL is developed, analyzed, and implemented to yield performance comparisons with value iteration and Dijkstra's algorithm using simple planning models. Next, mathematical analysis shows: 1) conditions under which cost minimization and reward maximization are equivalent, 2) conditions for equivalence of single-shot goal termination and infinite-horizon episodic learning, and 3) conditions under which discounting causes goal achievement to fail. The paper then advocates for defining and optimizing truecost, rather than inserting arbitrary parameters to guide operations. Performance studies are then extended to the stochastic case, using planning-oriented criteria and comparing value iteration to RL with learning rates and greediness factors.

📄 PDF Abstract BibTeX arXiv:2603.07844

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Decision Making

Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement Learning

2020-05-15 · Thomas M. Moerland, Anna Deichler, Simone Baldi, Joost Broekens 외

Planning and reinforcement learning are two key approaches to sequential decision making. Multi-step approximate real-time dynamic programming, a recently successful algorithm class of which AlphaZero [Silver et al., 201…

Decision Making Reinforcement Learning (RL)Sequential Decision Making

A New View on Planning in Online Reinforcement Learning

2024-06-03 · Kevin Roice, Parham Mohammad Panahi, Scott M. Jordan, Adam White 외

This paper investigates a new approach to model-based reinforcement learning using background planning: mixing (approximate) dynamic programming updates and model-free updates, similar to the Dyna architecture. Backgroun…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning

Planning and Learning: Path-Planning for Autonomous Vehicles, a Review of the Literature

2022-07-26 · Kevin Osanlou, Christophe Guettier, Tristan Cazenave, Eric Jacopin

This short review aims to make the reader familiar with state-of-the-art works relating to planning, scheduling and learning. First, we study state-of-the-art planning algorithms. We give a brief introduction of neural n…

Autonomous Vehicles reinforcement-learning Reinforcement Learning (RL)Scheduling

Action Categorization for Computationally Improved Task Learning and Planning

2018-04-26 · Lakshmi Nair, Sonia Chernova

This paper explores the problem of task learning and planning, contributing the Action-Category Representation (ACR) to improve computational performance of both Planning and Reinforcement Learning (RL). ACR is an algori…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Starcraft

Tight Regret Bounds for Model-Based Reinforcement Learning with Greedy Policies

2019-05-27 · NeurIPS 2019 12 · Yonathan Efroni, Nadav Merlis, Mohammad Ghavamzadeh, Shie Mannor

State-of-the-art efficient model-based Reinforcement Learning (RL) algorithms typically act by iteratively solving empirical models, i.e., by performing \emph{full-planning} on Markov Decision Processes (MDPs) built by t…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)