Data Valuation for Offline Reinforcement Learning
The success of deep reinforcement learning (DRL) hinges on the availability of training data, which is typically obtained via a large number of environment interactions. In many real-world scenarios, costs and risks are associated with gathering these data. The field of offline reinforcement learning addresses these issues through outsourcing the collection of data to a domain expert or a carefully monitored program and subsequently searching for a batch-constrained optimal policy. With the emergence of data markets, an alternative to constructing a dataset in-house is to purchase external data. However, while state-of-the-art offline reinforcement learning approaches have shown a lot of promise, they currently rely on carefully constructed datasets that are well aligned with the intended target domains. This raises questions regarding the transferability and robustness of an offline reinforcement learning agent trained on externally acquired data. In this paper, we empirically evaluate the ability of the current state-of-the-art offline reinforcement learning approaches to coping with the source-target domain mismatch within two MuJoCo environments, finding that current state-of-the-art offline reinforcement learning algorithms underperform in the target domain. To address this, we propose data valuation for offline reinforcement learning (DVORL), which allows us to identify relevant and high-quality transitions, improving the performance and transferability of policies learned by offline reinforcement learning algorithms. The results show that our method outperforms offline reinforcement learning baselines on two MuJoCo environments.
Code (0)
등록된 구현이 없습니다.
Tasks
Data ValuationDeep Reinforcement LearningMuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning
Value function estimation is an indispensable subroutine in reinforcement learning, which becomes more challenging in the offline setting. In this paper, we propose Hybrid Value Estimation (HVE) to reduce value estimatio…
MuJoCoOff-policy evaluationreinforcement-learningReinforcement Learning+1Offline Evaluation for Reinforcement Learning-based Recommendation: A Critical Issue and Some Alternatives
In this paper, we argue that the paradigm commonly adopted for offline evaluation of sequential recommender systems is unsuitable for evaluating reinforcement learning-based recommenders. We find that most of the existin…
Offline RLRecommendation Systemsreinforcement-learningReinforcement Learning+1Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbations
Offline reinforcement learning, which learns solely from datasets without environmental interaction, has gained attention. This approach, similar to traditional online deep reinforcement learning, is particularly promisi…
Deep Reinforcement LearningOpenAI Gymreinforcement-learningReinforcement LearningChallenges and Opportunities in Offline Reinforcement Learning from Visual Observations
Offline reinforcement learning has shown great promise in leveraging large pre-collected datasets for policy learning, allowing agents to forgo often-expensive online data collection. However, offline reinforcement learn…
Benchmarkingcontinuous-controlContinuous ControlOffline RL+3Importance of Empirical Sample Complexity Analysis for Offline Reinforcement Learning
We hypothesize that empirically studying the sample complexity of offline reinforcement learning (RL) is crucial for the practical applications of RL in the real world. Several recent works have demonstrated the ability …
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)