Data Valuation using Reinforcement Learning
Quantifying the value of data is a fundamental problem in machine learning. Data valuation has multiple important use cases: (1) building insights about the learning task, (2) domain adaptation, (3) corrupted sample discovery, and (4) robust learning. To adaptively learn data values jointly with the target task predictor model, we propose a meta learning framework which we name Data Valuation using Reinforcement Learning (DVRL). We employ a data value estimator (modeled by a deep neural network) to learn how likely each datum is used in training of the predictor model. We train the data value estimator using a reinforcement signal of the reward obtained on a small validation set that reflects performance on the target task. We demonstrate that DVRL yields superior data value estimates compared to alternative methods across different types of datasets and in a diverse set of application scenarios. The corrupted sample discovery performance of DVRL is close to optimal in many regimes (i.e. as if the noisy samples were known apriori), and for domain adaptation and robust learning DVRL significantly outperforms state-of-the-art by 14.6% and 10.8%, respectively.
Code (2)
Tasks
Data ValuationDomain AdaptationMeta-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Task 2Similar Papers 제목 키워드 기반
Comparing BERT-based Reward Functions for Deep Reinforcement Learning in Machine Translation
In text generation tasks such as machine translation, models are generally trained using cross-entropy loss. However, mismatches between the loss function and the evaluation metric are often problematic. It is known that…
Deep Reinforcement LearningMachine Translationreinforcement-learningReinforcement Learning+3Evaluation-Aware Reinforcement Learning
Policy evaluation is a core component of many reinforcement learning (RL) algorithms and a critical tool for ensuring safe deployment of RL policies. However, existing policy evaluation methods often suffer from high var…
Reinforcement LearningOffline Evaluation for Reinforcement Learning-based Recommendation: A Critical Issue and Some Alternatives
In this paper, we argue that the paradigm commonly adopted for offline evaluation of sequential recommender systems is unsuitable for evaluating reinforcement learning-based recommenders. We find that most of the existin…
Offline RLRecommendation Systemsreinforcement-learningReinforcement Learning+1Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms
Starting from the utilization of deep neural networks to approximate the state-action value function that led to winning one of the most challenging games, to algorithmic advancements that allowed solving problems withou…
Reinforcement LearningOnline Estimation and Inference for Robust Policy Evaluation in Reinforcement Learning
Recently, reinforcement learning has gained prominence in modern statistics, with policy evaluation being a key component. Unlike traditional machine learning literature on this topic, our work places emphasis on statist…
reinforcement-learningReinforcement Learning