Online Reinforcement Learning for Real-Time Exploration in Continuous State and Action Markov Decision Processes
This paper presents a new method to learn online policies in continuous state, continuous action, model-free Markov decision processes, with two properties that are crucial for practical applications. First, the policies are implementable with a very low computational cost: once the policy is computed, the action corresponding to a given state is obtained in logarithmic time with respect to the number of samples used. Second, our method is versatile: it does not rely on any a priori knowledge of the structure of optimal policies. We build upon the Fitted Q-iteration algorithm which represents the $Q$-value as the average of several regression trees. Our algorithm, the Fitted Policy Forest algorithm (FPF), computes a regression forest representing the Q-value and transforms it into a single tree representing the policy, while keeping control on the size of the policy using resampling and leaf merging. We introduce an adaptation of Multi-Resolution Exploration (MRE) which is particularly suited to FPF. We assess the performance of FPF on three classical benchmarks for reinforcement learning: the "Inverted Pendulum", the "Double Integrator" and "Car on the Hill" and show that FPF equals or outperforms other algorithms, although these algorithms rely on the use of particular representations of the policies, especially chosen in order to fit each of the three problems. Finally, we exhibit that the combination of FPF and MRE allows to find nearly optimal solutions in problems where $\epsilon$-greedy approaches would fail.
Code (0)
등록된 구현이 없습니다.
Tasks
regressionReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Active exploration in parameterized reinforcement learning
Online model-free reinforcement learning (RL) methods with continuous actions are playing a prominent role when dealing with real-world applications such as Robotics. However, when confronted to non-stationary environmen…
Meta-LearningQ-Learningreinforcement-learningReinforcement Learning+1Efficient Exploration in Continuous-time Model-based Reinforcement Learning
Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce a model-based reinforcement learning algorithm tha…
Efficient ExplorationGaussian ProcessesModel-based Reinforcement Learningreinforcement-learning+1Sample-efficient and Scalable Exploration in Continuous-Time RL
Reinforcement learning algorithms are typically designed for discrete-time dynamics, even though the underlying real-world control systems are often continuous in time. In this paper, we study the problem of continuous-t…
Reinforcement LearningGaussian ProcessesEvaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks
Deep reinforcement learning has been shown to solve challenging tasks where large amounts of training experience is available, usually obtained online while learning the task. Robotics is a significant potential applicat…
continuous-controlContinuous ControlDeep Reinforcement LearningReinforcement LearningExploration in Reinforcement Learning with Deep Covering Options
While many option discovery methods have been proposed to accelerate exploration in reinforcement learning, they are often heuristic. Recently, covering options was proposed to discover a set of options that provably red…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)