Augmenting Offline Reinforcement Learning with State-only Interactions
Batch offline data have been shown considerably beneficial for reinforcement learning. Their benefit is further amplified by upsampling with generative models. In this paper, we consider a novel opportunity where interaction with environment is feasible, but only restricted to observations, i.e., \textit{no reward} feedback is available. This setting is broadly applicable, as simulators or even real cyber-physical systems are often accessible, while in contrast reward is often difficult or expensive to obtain. As a result, the learner must make good sense of the offline data to synthesize an efficient scheme of querying the transition of state. Our method first leverages online interactions to generate high-return trajectories via conditional diffusion models. They are then blended with the original offline trajectories through a stitching algorithm, and the resulting augmented data can be applied generically to downstream reinforcement learners. Superior empirical performance is demonstrated over state-of-the-art data augmentation methods that are extended to utilize state-only interactions.
Code (0)
등록된 구현이 없습니다.
Tasks
D4RLData AugmentationKnowledge Distillationreinforcement-learningReinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis
This paper investigates a hybrid learning framework for reinforcement learning (RL) in which the agent can leverage both an offline dataset and online interactions to learn the optimal policy. We present a unified algori…
AllMulti-Armed BanditsReinforcement Learning (RL)GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
Offline Reinforcement Learning (Offline RL) presents challenges of learning effective decision-making policies from static datasets without any online interactions. Data augmentation techniques, such as noise injection a…
Data AugmentationDecision MakingOffline RLreinforcement-learning+1DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning
Offline reinforcement learning algorithms promise to be applicable in settings where a fixed dataset is available and no new experience can be acquired. However, such formulation is inevitably offline-data-hungry and, in…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)SOReL and TOReL: Two Methods for Fully Offline Reinforcement Learning
Sample efficiency remains a major obstacle for real world adoption of reinforcement learning (RL): success has been limited to settings where simulators provide access to essentially unlimited environment interactions, w…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Offline Reinforcement Learning with Fisher Divergence Critic Regularization
Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline d…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)