COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs
Counterfactual examples have proven to be valuable in the field of natural language processing (NLP) for both evaluating and improving the robustness of language models to spurious correlations in datasets. Despite their demonstrated utility for NLP, multimodal counterfactual examples have been relatively unexplored due to the difficulty of creating paired image-text data with minimal counterfactual changes. To address this challenge, we introduce a scalable framework for automatic generation of counterfactual examples using text-to-image diffusion models. We use our framework to create COCO-Counterfactuals, a multimodal counterfactual dataset of paired image and text captions based on the MS-COCO dataset. We validate the quality of COCO-Counterfactuals through human evaluations and show that existing multimodal models are challenged by our counterfactual image-text pairs. Additionally, we demonstrate the usefulness of COCO-Counterfactuals for improving out-of-domain generalization of multimodal vision-language models via training data augmentation.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualData AugmentationDomain GeneralizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evaluating and Mitigating Bias in Image Classifiers: A Causal Perspective Using Counterfactuals
Counterfactual examples for an input -- perturbations that change specific features but not others -- have been shown to be useful for evaluating bias of machine learning models, e.g., against specific demographic groups…
BIG-bench Machine LearningcounterfactualFairnessvalidCoCoGEC: Counterfactual Generation for Robust Grammatical Error Correction
Grammatical error correction (GEC) systems are usually trained and evaluated on GEC benchmarks, but their performance often drops sharply once the surrounding context is slightly perturbed or extended. This indicates tha…
Grammatical Error CorrectionData AugmentationCoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers
Dialogue state trackers have made significant progress on benchmark datasets, but their generalization capability to novel and realistic scenarios beyond the held-out conversations is less understood. We propose controll…
counterfactualDialogue State TrackingMulti-domain Dialogue State TrackingOutcome-Guided Counterfactuals for Reinforcement Learning Agents from a Jointly Trained Generative Latent Space
We present a novel generative method for producing unseen and plausible counterfactual examples for reinforcement learning (RL) agents based upon outcome variables that characterize agent behavior. Our approach uses a va…
counterfactualReinforcement Learning (RL)On Quantitative Evaluations of Counterfactuals
As counterfactual examples become increasingly popular for explaining decisions of deep learning models, it is essential to understand what properties quantitative evaluation metrics do capture and equally important what…
counterfactual