Online Decision Transformer
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling. However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via taskspecific interactions with the environment. We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in a unified framework. Our framework uses sequence-level entropy regularizers in conjunction with autoregressive modeling objectives for sample-efficient exploration and finetuning. Empirically, we show that ODT is competitive with the state-of-the-art in absolute performance on the D4RL benchmark but shows much more significant gains during the finetuning procedure.
Code (2)
Tasks
D4RLEfficient ExplorationLanguage ModelingLanguage ModellingReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing a trajectory in an autoregressive way. While improvements have been made to overcome initia…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Online Finetuning Decision Transformers with Pure RL Gradients
Decision Transformers (DTs) have emerged as a powerful framework for sequential decision making by formulating offline reinforcement learning (RL) as a sequence modeling problem. However, extending DTs to online settings…
Reinforcement LearningDecision MakingOnDeFog: Online Decision Transformer under Frame Dropping
In challenging real-world reinforcement learning applications, communication delays or sensor failures often cause frame dropping, in which the agent cannot receive the dropped states and associated rewards. To address t…
Reinforcement LearningMaximum-Entropy Regularized Decision Transformer with Reward Relabelling for Dynamic Recommendation
Reinforcement learning-based recommender systems have recently gained popularity. However, due to the typical limitations of simulation environments (e.g., data inefficiency), most of the work cannot be broadly applied i…
Recommendation Systemsreinforcement-learningReinforcement LearningDODT: Enhanced Online Decision Transformer Learning through Dreamer's Actor-Critic Trajectory Forecasting
Advancements in reinforcement learning have led to the development of sophisticated models capable of learning complex decision-making tasks. However, efficiently integrating world models with decision transformers remai…
Decision MakingModel-based Reinforcement Learningreinforcement-learningReinforcement Learning+1