Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement
We commonly use agreement measures to assess the utility of judgements made by human annotators in Natural Language Processing (NLP) tasks. While inter-annotator agreement is frequently used as an indication of label reliability by measuring consistency between annotators, we argue for the additional use of intra-annotator agreement to measure label stability over time. However, in a systematic review, we find that the latter is rarely reported in this field. Calculating these measures can act as important quality control and provide insights into why annotators disagree. We propose exploratory annotation experiments to investigate the relationships between these measures and perceptions of subjectivity and ambiguity in text items.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
JADE: Joint Autoencoders for Dis-Entanglement
The problem of feature disentanglement has been explored in the literature, for the purpose of image and video processing and text analysis. State-of-the-art methods for disentangling feature representations rely on the …
DisentanglementGeneral ClassificationSubjective $\textit{Isms}$? On the Danger of Conflating Hate and Offence in Abusive Language Detection
Natural language processing research has begun to embrace the notion of annotator subjectivity, motivated by variations in labelling. This approach understands each annotator's view as valid, which can be highly suitable…
Abusive LanguageHate Speech DetectionSentiment AnalysisvalidDisentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders
Generative models that learn disentangled representations for different factors of variation in an image can be very useful for targeted data augmentation. By sampling from the disentangled latent subspace of interest, w…
Data AugmentationVariational Semi-supervised Aspect-term Sentiment Analysis via Transformer
Aspect-term sentiment analysis (ATSA) is a longstanding challenge in natural language understanding. It requires fine-grained semantical reasoning about a target entity appeared in the text. As manual annotation over the…
Aspect-Based Sentiment Analysis (ABSA)Natural Language UnderstandingSentiment AnalysisVariational InferenceUnlabeled Disentangling of GANs with Guided Siamese Networks
Disentangling underlying generative factors of a data distribution is important for interpretability and generalizable representations. In this paper, we introduce two novel disentangling methods. Our first method, Unla…