A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control
Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design decisions such as how to structure the policy and value networks, share components, or handle temporal information. In this paper, we show that transformers can be strong baselines for continuous control in online model-free RL. We investigate key design questions: how to condition inputs, share components between actor and critic, and slice sequential data for training. Our experiments reveal stable architectural and training strategies enabling competitive performance across fully and partially observable tasks, and in both vector- and image-based settings. These findings offer practical guidance for applying transformers in online RL.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningContinuous ControlSimilar Papers 제목 키워드 기반
Mental Accounts for Actions: EWA-Inspired Attention in Decision Transformers
Transformers have emerged as a compelling architecture for sequential decision-making by modeling trajectories via self-attention. In reinforcement learning (RL), they enable return-conditioned control without relying on…
Reinforcement LearningThe Sample Complexity of Online Reinforcement Learning: A Multi-model Perspective
We study the sample complexity of online reinforcement learning in the general setting of nonlinear dynamical systems with continuous state and action spaces. Our analysis accommodates a large class of dynamical systems …
Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control
While Deep Reinforcement Learning (DRL) has emerged as a promising approach to many complex tasks, it remains challenging to train a single DRL agent that is capable of undertaking multiple different continuous control t…
continuous-controlContinuous ControlDeep Reinforcement LearningMuJoCo+4RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems
Nonlinear optimal control problems are often solved with numerical methods that require knowledge of system's dynamics which may be difficult to infer, and that carry a large computational cost associated with iterative …
continuous-controlContinuous ControlHierarchical Reinforcement LearningModel-based Reinforcement Learning+3Safe reinforcement learning control for continuous-time nonlinear systems without a backup controller
This paper proposes an on-policy reinforcement learning (RL) control algorithm that solves the optimal regulation problem for a class of uncertain continuous-time nonlinear systems under user-defined state constraints. W…
reinforcement-learningReinforcement Learning (RL)Safe Reinforcement Learning