Efficient Exploration in Continuous-time Model-based Reinforcement Learning

2023-10-30 · NeurIPS 2023 11

Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce a model-based reinforcement learning algorithm that represents continuous-time dynamics using nonlinear ordinary differential equations (ODEs). We capture epistemic uncertainty using well-calibrated probabilistic models, and use the optimistic principle for exploration. Our regret bounds surface the importance of the measurement selection strategy(MSS), since in continuous time we not only must decide how to explore, but also when to observe the underlying system. Our analysis demonstrates that the regret is sublinear when modeling ODEs with Gaussian Processes (GP) for common choices of MSS, such as equidistant sampling. Additionally, we propose an adaptive, data-dependent, practical MSS that, when combined with GP dynamics, also achieves sublinear regret with significantly fewer samples. We showcase the benefits of continuous-time modeling over its discrete-time counterpart, as well as our proposed adaptive MSS over standard baselines, on several applications.

📄 PDF Abstract BibTeX arXiv:2310.19848

Code (0)

등록된 구현이 없습니다.

Tasks

Efficient ExplorationGaussian ProcessesModel-based Reinforcement Learningreinforcement-learning Reinforcement Learning

Continuous-Time Reinforcement Learning for Asset-Liability Management

2025-09-27 · Yilie Huang arxiv

This paper proposes a novel approach for Asset-Liability Management (ALM) by employing continuous-time Reinforcement Learning (RL) with a linear-quadratic (LQ) formulation that incorporates both interim and terminal obje…

Reinforcement Learning

Sampling-Based Safe Reinforcement Learning

2026-05-19 · Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl 외 arxiv

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL alg…

Reinforcement Learning Continuous Control

k-Means Maximum Entropy Exploration

2022-05-31 · Alexander Nedergaard, Matthew Cook

Exploration in high-dimensional, continuous spaces with sparse rewards is an open problem in reinforcement learning. Artificial curiosity algorithms address this by creating rewards that lead to exploration. Given a rein…

Density Estimation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Discrete-Time Mean-Variance Strategy Based on Reinforcement Learning

2023-12-24 · Xiangyu Cui, Xun Li, Yun Shi, Si Zhao

This paper studies a discrete-time mean-variance model based on reinforcement learning. Compared with its continuous-time counterpart in \cite{zhou2020mv}, the discrete-time model makes more general assumptions about the…

reinforcement-learning Reinforcement Learning

Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems

2025-07-01 · Yilie Huang, Xun Yu Zhou arxiv

We study reinforcement learning (RL) for the same class of continuous-time stochastic linear--quadratic (LQ) control problems as in \cite{huang2024sublinear}, where volatilities depend on both states and controls while s…