Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets
Natural language processing models often exploit spurious correlations between task-independent features and labels in datasets to perform well only within the distributions they are trained on, while not generalising to different task distributions. We propose to tackle this problem by generating a debiased version of a dataset, which can then be used to train a debiased, off-the-shelf model, by simply replacing its training data. Our approach consists of 1) a method for training data generators to generate high-quality, label-consistent data samples; and 2) a filtering mechanism for removing data points that contribute to spurious correlations, measured in terms of z-statistics. We generate debiased versions of the SNLI and MNLI datasets, and we evaluate on a large suite of debiased, out-of-distribution, and adversarial test sets. Results show that models trained on our debiased datasets generalise better than those trained on the original datasets in all settings. On the majority of the datasets, our method outperforms or performs comparably to previous state-of-the-art debiasing strategies, and when combined with an orthogonal technique, product-of-experts, it improves further and outperforms previous best results of SNLI-hard and MNLI-hard.
Code (1)
Tasks
Natural Language InferenceSimilar Papers 제목 키워드 기반
Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets
Natural language processing models often exploit spurious correlations between task-independent features and labels in datasets to perform well only within the distributions they are trained on, while not generalising to…
Natural Language InferenceLess is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery
In this paper, we conduct the first study on spurious correlations for open-domain response generation models based on a corpus CGDIALOG curated in our work. The cur rent models indeed suffer from spurious correlations a…
Causal DiscoveryInformativenessResponse GenerationSpuriousness-Aware Meta-Learning for Learning Robust Classifiers
Spurious correlations are brittle associations between certain attributes of inputs and target variables, such as the correlation between an image background and an object class. Deep image classifiers often leverage the…
AttributeLanguage ModellingMeta-LearningDecorrelate Irrelevant, Purify Relevant: Overcome Textual Spurious Correlations from a Feature Perspective
Natural language understanding (NLU) models tend to rely on spurious correlations (i.e., dataset bias) to achieve high performance on in-distribution datasets but poor performance on out-of-distribution ones. Most of the…
Fact VerificationNatural Language InferenceNatural Language UnderstandingFocusing Image Generation to Mitigate Spurious Correlations
Instance features in images exhibit spurious correlations with background features, affecting the training process of deep neural classifiers. This leads to insufficient attention to instance features by the classifier, …
AttributeData AugmentationDiversityImage Generation