Adaptive Discretization for Model-Based Reinforcement Learning

2020-07-01 · NeurIPS 2020 12 · Sean R. Sinclair, Tianyu Wang, Gauri Jain, Siddhartha Banerjee, Christina Lee Yu

We introduce the technique of adaptive discretization to design an efficient model-based episodic reinforcement learning algorithm in large (potentially continuous) state-action spaces. Our algorithm is based on optimistic one-step value iteration extended to maintain an adaptive discretization of the space. From a theoretical perspective we provide worst-case regret bounds for our algorithm which are competitive compared to the state-of-the-art model-based algorithms. Moreover, our bounds are obtained via a modular proof technique which can potentially extend to incorporate additional structure on the problem. From an implementation standpoint, our algorithm has much lower storage and computational requirements due to maintaining a more efficient partition of the state and action spaces. We illustrate this via experiments on several canonical control problems, which shows that our algorithm empirically performs significantly better than fixed discretization in terms of both faster convergence and lower memory usage. Interestingly, we observe empirically that while fixed-discretization model-based algorithms vastly outperform their model-free counterparts, the two achieve comparable performance with adaptive discretization.

📄 PDF Abstract BibTeX arXiv:2007.00717

Code (1)

seanrsinclair/AdaptiveQLearning 공식 구현

Tasks

modelModel-based Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Adaptive Parameter Selection in Evolutionary Algorithms by Reinforcement Learning with Dynamic Discretization of Parameter Range

2016-03-22 · Arkady Rost, Irina Petrova, Arina Buzdalova

Online parameter controllers for evolutionary algorithms adjust values of parameters during the run of an evolutionary algorithm. Recently a new efficient parameter controller based on reinforcement learning was proposed…

Evolutionary Algorithms reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Adaptive Discretization for Episodic Reinforcement Learning in Metric Spaces

2019-10-17 · Sean R. Sinclair, Siddhartha Banerjee, Christina Lee Yu

We present an efficient algorithm for model-free episodic reinforcement learning on large (potentially continuous) state-action spaces. Our algorithm is based on a novel $Q$-learning policy with adaptive data-driven disc…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization

2022-02-02 · Dianbo Liu, Alex Lamb, Xu Ji, Pascal Notsawo 외

Vector Quantization (VQ) is a method for discretizing latent representations and has become a major part of the deep learning toolkit. It has been theoretically and empirically shown that discretization of representation…

Quantization reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Deep Reinforcement Learning for Adaptive Mesh Refinement

2022-09-25 · Corbin Foucart, Aaron Charous, Pierre F. J. Lermusiaux

Finite element discretizations of problems in computational physics often rely on adaptive mesh refinement (AMR) to preferentially resolve regions containing important features during simulation. However, these spatial r…

Decision Making Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+2

Adaptive Discretization in Online Reinforcement Learning

2021-10-29 · Sean R. Sinclair, Siddhartha Banerjee, Christina Lee Yu

Discretization based approaches to solving online reinforcement learning problems have been studied extensively in practice on applications ranging from resource allocation to cache management. Two major questions in des…

Management reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)