An Investigation of the (In)effectiveness of Counterfactually Augmented Data
While pretrained language models achieve excellent performance on natural language understanding benchmarks, they tend to rely on spurious correlations and generalize poorly to out-of-distribution (OOD) data. Recent work has explored using counterfactually-augmented data (CAD) -- data generated by minimally perturbing examples to flip the ground-truth label -- to identify robust features that are invariant under distribution shift. However, empirical results using CAD for OOD generalization have been mixed. To explain this discrepancy, we draw insights from a linear Gaussian model and demonstrate the pitfalls of CAD. Specifically, we show that (a) while CAD is effective at identifying robust features, it may prevent the model from learning unperturbed robust features; and (b) CAD may exacerbate existing spurious correlations in the data. On two crowdsourced CAD datasets, our results show that the lack of perturbation diversity limits their effectiveness on OOD generalization, calling for innovative crowdsourcing procedures to elicit diverse perturbation of examples.
Code (1)
Tasks
DiversityNatural Language UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An Investigation of the (In)effectiveness of Counterfactually Augmented Data
While pretrained language models achieve excellent performance on natural language understanding benchmarks, they tend to rely on spurious correlations and generalize poorly to out-of-distribution (OOD) data. Recent work…
DiversityNatural Language UnderstandingCounterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data
A growing body of work shows that models exploit annotation artifacts to achieve state-of-the-art performance on standard crowdsourced benchmarks---datasets collected from crowdworkers to create an evaluation task---whil…
counterfactualNatural Language InferenceNatural Language UnderstandingReading ComprehensionRethinking Video-Text Understanding: Retrieval from Counterfactually Augmented Data
Recent video-text foundation models have demonstrated strong performance on a wide variety of downstream video understanding tasks. Can these video-text models genuinely understand the contents of natural videos? Standar…
Language ModellingLarge Language ModelRetrievalVideo UnderstandingCounterfactually Augmented Data and Unintended Bias: The Case of Sexism and Hate Speech Detection
Counterfactually Augmented Data (CAD) aims to improve out-of-domain generalizability, an indicator of model robustness. The improvement is credited with promoting core features of the construct over spurious artifacts th…
Hate Speech DetectionPairCFR: Enhancing Model Training on Paired Counterfactually Augmented Data through Contrastive Learning
Counterfactually Augmented Data (CAD) involves creating new data samples by applying minimal yet sufficient modifications to flip the label of existing data samples to other classes. Training with CAD enhances model robu…
Contrastive Learningcounterfactual