Policy Transfer for Continuous-Time Reinforcement Learning: A (Rough) Differential Equation Approach
This paper studies policy transfer, one of the well-known transfer learning techniques adopted in large language models, for continuous-time reinforcement learning problems. In the case of continuous-time linear-quadratic systems with Shannon's entropy regularization, we fully exploit the Gaussian structure of their optimal policy and the stability of their associated Riccati equations. In the general case where the system has possibly non-linear and bounded dynamics, the key technical component is the stability of diffusion SDEs which is established by invoking the rough path theory. Our work provides the first theoretical proof of policy transfer for continuous-time RL: an optimal policy learned for one RL problem can be used to initialize to search for a near-optimal policy for another closely related RL problem, while achieving (at least) the same rate of convergence for the original algorithm. As a byproduct of our analysis, we derive the stability of a concrete class of continuous-time score-based diffusion models via their connection with LQRs. To illustrate the benefit of policy transfer for RL, we propose a novel policy learning algorithm for continuous-time LQRs, which achieves global linear convergence and local super-linear convergence.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningTransfer LearningSimilar Papers 제목 키워드 기반
Transfer Value or Policy? A Value-centric Framework Towards Transferrable Continuous Reinforcement Learning
Transferring learned knowledge from one environment to another is an important step towards practical reinforcement learning (RL). In this paper, we investigate the problem of transfer learning across environments with d…
continuous-controlContinuous ControlReinforcement Learning (RL)Transfer LearningMeta-Evolve: Continuous Robot Evolution for One-to-many Policy Transfer
We investigate the problem of transferring an expert policy from a source robot to multiple different robots. To solve this problem, we propose a method named $Meta$-$Evolve$ that uses continuous robot evolution to effic…
Transferring Domain Knowledge with an Adviser in Continuous Tasks
Recent advances in Reinforcement Learning (RL) have surpassed human-level performance in many simulated environments. However, existing reinforcement learning techniques are incapable of explicitly incorporating already …
OpenAI Gymreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedule
We consider time discretization for score-based diffusion models to generate samples from a learned reverse-time dynamic on a finite grid. Uniform and hand-crafted grids can be suboptimal given a budget on the number of …
Reinforcement LearningPolicy Optimization for Continuous Reinforcement Learning
We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon r…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)