Ecologically Valid Explanations for Label Variation in NLI

2023-10-20 · Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe

Human label variation, or annotation disagreement, exists in many natural language processing (NLP) tasks, including natural language inference (NLI). To gain direct evidence of how NLI label variation arises, we build LiveNLI, an English dataset of 1,415 ecologically valid explanations (annotators explain the NLI labels they chose) for 122 MNLI items (at least 10 explanations per item). The LiveNLI explanations confirm that people can systematically vary on their interpretation and highlight within-label variation: annotators sometimes choose the same label for different reasons. This suggests that explanations are crucial for navigating label interpretations in general. We few-shot prompt large language models to generate explanations but the results are inconsistent: they sometimes produces valid and informative explanations, but it also generates implausible ones that do not support the label, highlighting directions for improvement.

📄 PDF Abstract BibTeX arXiv:2310.13850

Code (1)

njjiang/livenli 공식 구현

Tasks

Natural Language Inferencevalid

Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation

2023-04-24 · Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe

Human label variation (Plank 2022), or annotation disagreement, exists in many natural language processing (NLP) tasks. To be robust and trusted, NLP models need to identify such variation and be able to explain it. To t…

In-Context Learning Natural Language Inference valid

WildfireGenome: Interpretable Machine Learning Reveals Local Drivers of Wildfire Risk and Their Cross-County Variation

2025-10-21 · Chenyue Liu, Ali Mostafavi arxiv

Current wildfire risk assessments rely on coarse hazard maps and opaque machine learning models that optimize regional accuracy while sacrificing interpretability at the decision scale. WildfireGenome addresses these gap…

Interpretable Machine Learning

LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference

2025-05-28 · Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe 외

There is increasing evidence of Human Label Variation (HLV) in Natural Language Inference (NLI), where annotators assign different labels to the same premise-hypothesis pair. However, within-label variation--cases where …

Explanation Generation Natural Language Inference

EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI

2025-11-12 · Longfei Zuo, Barbara Plank, Siyao Peng arxiv

High-quality datasets are critical for training and evaluating reliable NLP models. In tasks like natural language inference (NLI), human label variation (HLV) arises when multiple labels are valid for the same instance,…

Natural Language Inference Explanation Generation

Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations

2025-10-18 · Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe 외 arxiv

Natural Language Inference (NLI) datasets often exhibit human label variation. To better understand these variations, explanation-based approaches analyze the underlying reasoning behind annotators' decisions. One such a…

Natural Language Inference Semantic Similarity