Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
Reinforcement learning (RL) algorithms usually require a substantial amount of interaction data and perform well only for specific tasks in a fixed environment. In some scenarios such as healthcare, however, usually only few records are available for each patient, and patients may show different responses to the same treatment, impeding the application of current RL algorithms to learn optimal policies. To address the issues of mechanism heterogeneity and related data scarcity, we propose a data-efficient RL algorithm that exploits structural causal models (SCMs) to model the state dynamics, which are estimated by leveraging both commonalities and differences across subjects. The learned SCM enables us to counterfactually reason what would have happened had another treatment been taken. It helps avoid real (possibly risky) exploration and mitigates the issue that limited experiences lead to biased policies. We propose counterfactual RL algorithms to learn both population-level and individual-level policies. We show that counterfactual outcomes are identifiable under mild conditions and that Q- learning on the counterfactual-based augmented data set converges to the optimal value function. Experimental results on synthetic and real-world data demonstrate the efficacy of the proposed approach.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualData AugmentationQ-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Counterfactual Data Augmentation using Locally Factored Dynamics
Many dynamic processes, including common scenarios in robotic control and reinforcement learning (RL), involve a set of interacting subprocesses. Though the subprocesses are not independent, their interactions are often …
counterfactualData AugmentationGeneral Reinforcement LearningMulti-Goal Reinforcement Learning+4Implicit Counterfactual Data Augmentation for Robust Learning
Machine learning models are prone to capturing the spurious correlations between non-causal attributes and classes, with counterfactual data augmentation being a promising direction for breaking these spurious associatio…
counterfactualData AugmentationMeta-LearningOut-of-Distribution GeneralizationCausalDyna: Improving Generalization of Dyna-style Reinforcement Learning via Counterfactual-Based Data Augmentation
Deep reinforcement learning agents trained in real-world environments with a limited diversity of object properties to learn manipulation tasks tend to suffer overfitting and fail to generalize to unseen testing environm…
counterfactualData AugmentationDeep Reinforcement LearningDiversity+4Reinforced Counterfactual Data Augmentation for Dual Sentiment Classification
Data augmentation and adversarial perturbation approaches have recently achieved promising results in solving the over-fitting problem in many natural language processing (NLP) tasks including sentiment classification. H…
ClassificationcounterfactualData AugmentationSentiment Analysis+2Efficient Classification with Counterfactual Reasoning and Active Learning
Data augmentation is one of the most successful techniques to improve the classification accuracy of machine learning models in computer vision. However, applying data augmentation to tabular data is a challenging proble…
Active LearningClassificationcounterfactualCounterfactual Reasoning+1