Backstepping Temporal Difference Learning
Off-policy learning ability is an important feature of reinforcement learning (RL) for practical applications. However, even one of the most elementary RL algorithms, temporal-difference (TD) learning, is known to suffer form divergence issue when the off-policy scheme is used together with linear function approximation. To overcome the divergent behavior, several off-policy TD-learning algorithms, including gradient-TD learning (GTD), and TD-learning with correction (TDC), have been developed until now. In this work, we provide a unified view of such algorithms from a purely control-theoretic perspective, and propose a new convergent algorithm. Our method relies on the backstepping technique, which is widely used in nonlinear control theory. Finally, convergence of the proposed algorithm is experimentally verified in environments where the standard TD-learning is known to be unstable.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement Learning (RL)Similar Papers 제목 키워드 기반
Gain-Only Neural Operators for PDE Backstepping
In this work we advance the recently-introduced deep learning-powered approach to PDE backstepping control by proposing a method that approximates only the control gain function -- a function of one variable -- instead o…
SchedulingBackstepping Design Embedded With Time-Varying Command Filters
If embedded with command filter properly, the implementation of backstepping design could be dramatically simplified. In this paper, we introduce a command filter with time-varying gain and integrate it with backstepping…
Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay
Control of distributed parameter systems affected by delays is a challenging task, particularly when the delays depend on spatial variables. The idea of integrating analytical control theory with learning-based control w…
Deep Reinforcement LearningReinforcement Learning (RL)Backstepping Control of Coupled General Hyperbolic-Parabolic PDE-PDE Systems
This paper considers the backstepping state feedback and observer design for hyperbolic and parabolic PDEs, which are bidirectionally interconnected in a general coupling structure. Both PDE subsystems consist of coupled…
PDE-based multi-agent formation control using flatness and backstepping: analysis, design and robot experiments
A PDE-based control concept is developed to deploy a multi-agent system into desired formation profiles. The dynamic model is based on a coupled linear, time-variant parabolic distributed parameter system. By means of a …
Motion Planning