Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering

2022-10-26 · Qingyi Si, Yuanxin Liu, Zheng Lin, Peng Fu, Weiping Wang

Despite the excellent performance of vision-language pre-trained models (VLPs) on conventional VQA task, they still suffer from two problems: First, VLPs tend to rely on language biases in datasets and fail to generalize to out-of-distribution (OOD) data. Second, they are inefficient in terms of memory footprint and computation. Although promising progress has been made in both problems, most existing works tackle them independently. To facilitate the application of VLP to VQA tasks, it is imperative to jointly study VLP compression and OOD robustness, which, however, has not yet been explored. This paper investigates whether a VLP can be compressed and debiased simultaneously by searching sparse and robust subnetworks. To this end, we systematically study the design of a training and compression pipeline to search the subnetworks, as well as the assignment of sparsity to different modality-specific modules. Our experiments involve 3 VLPs, 2 compression methods, 4 training methods, 2 datasets and a range of sparsity levels and random seeds. Our results show that there indeed exist sparse and robust subnetworks, which are competitive with the debiased full VLP and clearly outperform the debiasing SoTAs with fewer parameters on OOD datasets VQA-CP v2 and VQA-VS. The codes can be found at https://github.com/PhoebusSi/Compress-Robust-VQA.

📄 PDF Abstract BibTeX arXiv:2210.14558

Code (1)

phoebussi/compress-robust-vqa 공식 구현 pytorch

Tasks

Question Answering Visual Question Answering Visual Question Answering (VQA)

Methods 이 논문이 사용한 방법론

fail 설명 없음

LXMERT LXMERT is a model for learning vision-and-language cross-modality representations. It consists of a Transformer model that consists three encoders: object relationship encoder, a…

SegDebias: Test-Time Bias Mitigation for ViT-Based CLIP via Segmentation

2025-11-01 · Fangyu Wu, Yujun Cai arxiv

Vision language models such as CLIP have shown remarkable performance in zero shot classification, but remain susceptible to spurious correlations, where irrelevant visual features influence predictions. Existing debiasi…

Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)

2024-10-25 · Leander Girrbach, Stephan Alaniz, Yiran Huang, Trevor Darrell 외

Pre-trained large language models (LLMs) have been reliably integrated with visual input for multimodal tasks. The widespread adoption of instruction-tuned image-to-text vision-language assistants (VLAs) like LLaVA and I…

Attribute Image to text

BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models

2026-01-05 · Sunny Gupta, Shounak Das, Amit Sethi arxiv

Vision language foundation models such as CLIP exhibit impressive zero-shot generalization yet remain vulnerable to spurious correlations across visual and textual modalities. Existing debiasing approaches often address …

Zero-shot Generalization Test-time Adaptation

A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning

2022-03-22 · Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang 외

Vision-language models can encode societal biases and stereotypes, but there are challenges to measuring and mitigating these multimodal harms due to lacking measurement robustness and feature degradation. To address the…

DeAR: Debiasing Vision-Language Models with Additive Residuals

2023-03-18 · CVPR 2023 1 · Ashish Seth, Mayur Hemani, Chirag Agarwal

Large pre-trained vision-language models (VLMs) reduce the time for developing predictive models for various vision-grounded language downstream tasks by providing rich, adaptable image and text representations. However,…

Attribute Benchmarking Fairness TAG