Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition
Object-context shortcuts remain a persistent challenge in vision-language models, undermining zero-shot reliability when test-time scenes differ from familiar training co-occurrences. We recast this issue as a causal inference problem and ask: Would the prediction remain if the object appeared in a different environment? To answer this at inference time, we estimate object and background expectations within CLIP's representation space, and synthesize counterfactual embeddings by recombining object features with diverse alternative contexts sampled from external datasets, batch neighbors, or text-derived descriptions. By estimating the Total Direct Effect and simulating intervention, we further subtract background-only activation, preserving beneficial object-context interactions while mitigating hallucinated scores. Without retraining or prompt design, our method substantially improves both worst-group and average accuracy on context-sensitive benchmarks, establishing a new zero-shot state of the art. Beyond performance, our framework provides a lightweight representation-level counterfactual approach, offering a practical causal avenue for debiased and reliable multimodal reasoning.
Code (0)
등록된 구현이 없습니다.
Tasks
Multimodal ReasoningCausal InferenceSimilar Papers 제목 키워드 기반
Jointly Optimizing Debiased CTR and Uplift for Coupons Marketing: A Unified Causal Framework
In online advertising, marketing interventions such as coupons introduce significant confounding bias into Click-Through Rate (CTR) prediction. Observed clicks reflect a mixture of users' intrinsic preferences and the up…
Debiased Learning from Naturally Imbalanced Pseudo-Labels
Pseudo-labels are confident predictions made on unlabeled target data by a classifier trained on labeled source data. They are widely used for adapting a model to unlabeled data, e.g., in a semi-supervised learning setti…
counterfactualCounterfactual ReasoningFew-Shot Image Classificationimbalanced classification+2Automatic, Debiased, and Invariant Counterfactual Generation under General Interventions
Generative models for counterfactual outcomes have great potential to support decision-making under complex interventions, but existing approaches are limited by unstable estimation, poor generalization across environmen…
Debiased Counterfactual Generation via Flow Matching from Observations
Estimating counterfactual distributions under interventions is central to treatment risk assessment and counterfactual generation tasks. Existing approaches model the counterfactual distribution as a standalone generativ…
Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration
Stance detection is critical for understanding the underlying position or attitude expressed toward a topic. Large language models (LLMs) have demonstrated significant advancements across various natural language process…
counterfactualDomain GeneralizationStance DetectionZero-Shot Stance Detection