Annotation Artifacts in Natural Language Inference Data
Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et. al, 2015) and 53% of MultiNLI (Williams et. al, 2017). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceNegationSentenceText CategorizationSimilar Papers 제목 키워드 기반
Mitigating Annotation Artifacts in Natural Language Inference Datasets to Improve Cross-dataset Generalization Ability
Natural language inference (NLI) aims at predicting the relationship between a given pair of premise and hypothesis. However, several works have found that there widely exists a bias pattern called annotation artifacts i…
Natural Language InferenceHypothesis-only Biases in Large Language Model-Elicited Natural Language Inference
We test whether replacing crowdsource workers with LLMs to write Natural Language Inference (NLI) hypotheses similarly results in annotation artifacts. We recreate a portion of the Stanford NLI corpus using GPT-4, Llama-…
Language ModelingLanguage ModellingLarge Language ModelNatural Language InferenceAugmenting NLP data to counter Annotation Artifacts for NLI Tasks
In this paper, we explore Annotation Artifacts - the phenomena wherein large pre-trained NLP models achieve high performance on benchmark datasets but do not actually "solve" the underlying task and instead rely on some …
Data AugmentationNatural Language InferenceReliable Evaluations for Natural Language Inference based on a Unified Cross-dataset Benchmark
Recent studies show that crowd-sourced Natural Language Inference (NLI) datasets may suffer from significant biases like annotation artifacts. Models utilizing these superficial clues gain mirage advantages on the in-dom…
Natural Language InferenceSeveral Experiments on Investigating Pretraining and Knowledge-Enhanced Models for Natural Language Inference
Natural language inference (NLI) is among the most challenging tasks in natural language understanding. Recent work on unsupervised pretraining that leverages unsupervised signals such as language-model and sentence pred…
Language ModelingLanguage ModellingNatural Language InferenceNatural Language Understanding+1