A Temporal Difference Method for Stochastic Continuous Dynamics
For continuous systems modeled by dynamical equations such as ODEs and SDEs, Bellman's principle of optimality takes the form of the Hamilton-Jacobi-Bellman (HJB) equation, which provides the theoretical target of reinforcement learning (RL). Although recent advances in RL successfully leverage this formulation, the existing methods typically assume the underlying dynamics are known a priori because they need explicit access to the coefficient functions of dynamical equations to update the value function following the HJB equation. We address this inherent limitation of HJB-based RL; we propose a model-free approach still targeting the HJB equation and propose the corresponding temporal difference method. We demonstrate its potential advantages over transition kernel-based formulations, both qualitatively and empirically. The proposed formulation paves the way toward bridging stochastic optimal control and model-free reinforcement learning.
Code (1)
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise
Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asympt…
A Spatio-temporal Continuous Network for Stochastic 3D Human Motion Prediction
Stochastic Human Motion Prediction (HMP) has received increasing attention due to its wide applications. Despite the rapid progress in generative fields, existing methods often face challenges in learning continuous temp…
Set-Valued Risk Measures as Backward Stochastic Difference Inclusions and Equations
Scalar dynamic risk measures for univariate positions in continuous time are commonly represented as backward stochastic differential equations. In the multivariate setting, dynamic risk measures have been defined and st…
Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features
We investigate the test risk of continuous-time stochastic gradient flow dynamics in learning theory. Using a path integral formulation we provide, in the regime of a small learning rate, a general formula for computing …
Learning TheoryTemporal Difference Learning with Continuous Time and State in the Stochastic Setting
We consider the problem of continuous-time policy evaluation. This consists in learning through observations the value function associated with an uncontrolled continuous-time stochastic dynamic and a reward function. We…
reinforcement-learningStochastic Optimization