Costate-focused models for reinforcement learning
Many recent algorithms for reinforcement learning are model-free and founded on the Bellman equation. Here we present a method founded on the costate equation and models of the state dynamics. We use the costate -- the gradient of cost with respect to state -- to improve the policy and also to "focus" the model, training it to detect and mimic those features of the environment that are most relevant to its task. We show that this method can handle difficult time-optimal control problems, driving deterministic or stochastic mechanical systems quickly to a target. On these tasks it works well compared to deep deterministic policy gradient, a recent Bellman method. And because it creates a model, the costate method can also learn from mental practice.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Value-Gradient Hypothesis of RL for LLMs
Reinforcement learning substantially improves pretrained language models, but it remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when they should provide the largest gains. …
Reinforcement LearningPontryagin Neural Operator for Solving Parametric General-Sum Differential Games
The values of two-player general-sum differential games are viscosity solutions to Hamilton-Jacobi-Isaacs (HJI) equations. Value and policy approximations for such games suffer from the curse of dimensionality (CoD). All…
Self-supervised diffusion model fine-tuning for costate initialization using Markov chain Monte Carlo
Global search and optimization of long-duration, low-thrust spacecraft trajectories with the indirect method is challenging due to a complex solution space and the difficulty of generating good initial guesses for the co…
Global Search for Optimal Low Thrust Spacecraft Trajectories using Diffusion Models and the Indirect Method
Long time-duration low-thrust nonlinear optimal spacecraft trajectory global search is a computationally and time expensive problem characterized by clustering patterns in locally optimal solutions. During preliminary mi…
Solve the General Constrained Optimal Control Problem with Common Integration Method
Computation of general state- and/or control-constrained Optimal Control Problems (OCPs) is difficult for various constraints, especially the intractable path constraint. For such problems, the theoretical convergence of…
Numerical Integration