Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions
Annotating data via crowdsourcing is time-consuming and expensive. Due to these costs, dataset creators often have each annotator label only a small subset of the data. This leads to sparse datasets with examples that are marked by few annotators. The downside of this process is that if an annotator doesn't get to label a particular example, their perspective on it is missed. This is especially concerning for subjective NLP datasets where there is no single correct label: people may have different valid opinions. Thus, we propose using imputation methods to generate the opinions of all annotators for all examples, creating a dataset that does not leave out any annotator's view. We then train and prompt models, using data from the imputed dataset, to make predictions about the distribution of responses and individual annotations. In our analysis of the results, we found that the choice of imputation method significantly impacts soft label changes and distribution. While the imputation introduces noise in the prediction of the original dataset, it has shown potential in enhancing shots for prompts, particularly for low-response-rate annotators. We have made all of our code and data publicly available.
Code (1)
Tasks
ImputationvalidSimilar Papers 제목 키워드 기반
Temporal Variational Implicit Neural Representations
We introduce Temporal Variational Implicit Neural Representations (TV-INRs), a probabilistic framework for modeling irregular multivariate time series that enables efficient individualized imputation and forecasting. By …
ImputationMeta-LearningTime SeriesA Convexified Matching Approach to Imputation and Individualized Inference
We introduce a new convexified matching method for missing value imputation and individualized inference inspired by computational optimal transport. Our method integrates favorable features from mainstream imputation ap…
counterfactualImputationImputation for prediction: beware of diminishing returns
Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. In this context, imputation is a common practice, driven by the hope that accurate imputations will enha…
ImputationMissing ValuesPredictionMixture-based Multiple Imputation Model for Clinical Data with a Temporal Dimension
The problem of missing values in multivariable time series is a key challenge in many applications such as clinical data mining. Although many imputation methods show their effectiveness in many applications, few of them…
Gaussian ProcessesImputationMissing ValuesTime Series+1FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning
Our proposed system FAMULUS helps students learn to diagnose based on automatic feedback in virtual patient simulations, and it supports instructors in labeling training data. Diagnosing is an exceptionally difficult ski…
DiagnosticMultiple-choice