paper-with-me

홈 › Papers

Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions

2023-05-24 · London Lowmanstone, Ruyuan Wan, Risako Owan, Jaehyung Kim, Dongyeop Kang

Annotating data via crowdsourcing is time-consuming and expensive. Due to these costs, dataset creators often have each annotator label only a small subset of the data. This leads to sparse datasets with examples that are marked by few annotators. The downside of this process is that if an annotator doesn't get to label a particular example, their perspective on it is missed. This is especially concerning for subjective NLP datasets where there is no single correct label: people may have different valid opinions. Thus, we propose using imputation methods to generate the opinions of all annotators for all examples, creating a dataset that does not leave out any annotator's view. We then train and prompt models, using data from the imputed dataset, to make predictions about the distribution of responses and individual annotations. In our analysis of the results, we found that the choice of imputation method significantly impacts soft label changes and distribution. While the imputation introduces noise in the prediction of the original dataset, it has shown potential in enhancing shots for prompts, particularly for low-response-rate annotators. We have made all of our code and data publicly available.

📄 PDF Abstract BibTeX arXiv:2305.15070

Code (1)

minnesotanlp/annotation-imputation 공식 구현 pytorch

Tasks

Imputationvalid

Similar Papers 제목 키워드 기반

Temporal Variational Implicit Neural Representations

2025-06-02 · Batuhan Koyuncu, Rachael DeVries, Ole Winther, Isabel Valera

We introduce Temporal Variational Implicit Neural Representations (TV-INRs), a probabilistic framework for modeling irregular multivariate time series that enables efficient individualized imputation and forecasting. By …

ImputationMeta-LearningTime Series

A Convexified Matching Approach to Imputation and Individualized Inference

2024-07-07 · YoonHaeng Hur, Tengyuan Liang

We introduce a new convexified matching method for missing value imputation and individualized inference inspired by computational optimal transport. Our method integrates favorable features from mainstream imputation ap…

counterfactualImputation

Imputation for prediction: beware of diminishing returns

2024-07-29 · Marine Le Morvan, Gaël Varoquaux

Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. In this context, imputation is a common practice, driven by the hope that accurate imputations will enha…

ImputationMissing ValuesPrediction

Mixture-based Multiple Imputation Model for Clinical Data with a Temporal Dimension

2019-08-12 · Ye Xue, Diego Klabjan, Yuan Luo

The problem of missing values in multivariable time series is a key challenge in many applications such as clinical data mining. Although many imputation methods show their effectiveness in many applications, few of them…

Gaussian ProcessesImputationMissing ValuesTime Series+1

FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning

2019-08-29 · IJCNLP 2019 11 · Jonas Pfeiffer, Christian M. Meyer, Claudia Schulz, Jan Kiesewetter 외

Our proposed system FAMULUS helps students learn to diagnose based on automatic feedback in virtual patient simulations, and it supports instructors in labeling training data. Diagnosing is an exceptionally difficult ski…

DiagnosticMultiple-choice