How (not) to ensemble LVLMs for VQA
This paper studies ensembling in the era of Large Vision-Language Models (LVLMs). Ensembling is a classical method to combine different models to get increased performance. In the recent work on Encyclopedic-VQA the authors examine a wide variety of models to solve their task: from vanilla LVLMs, to models including the caption as extra context, to models augmented with Lens-based retrieval of Wikipedia pages. Intuitively these models are highly complementary, which should make them ideal for ensembling. Indeed, an oracle experiment shows potential gains from 48.8% accuracy (the best single model) all the way up to 67% (best possible ensemble). So it is a trivial exercise to create an ensemble with substantial real gains. Or is it?
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Improving Generalization in Visual Reasoning via Self-Ensemble
The cognitive faculty of visual reasoning necessitates the integration of multimodal perceptual processing and commonsense and external knowledge of the world. In recent years, a plethora of large vision-language models …
Visual Question Answering (VQA)Visual ReasoningCARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Models
Large vision-language models (LVLMs) are typically trained using autoregressive language modeling objectives, which align visual representations with linguistic space. While effective for multimodal reasoning, this align…
Multimodal ReasoningImage ClassificationUnlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection
Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, …
Visual Question AnsweringImage ClassificationImage CaptioningTABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs
Speculative decoding (SD) has proven effective for accelerating LLM inference by quickly generating draft tokens and verifying them in parallel. However, SD remains largely unexplored for Large Vision-Language Models (LV…
Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
Large Vision-Language Models (LVLMs) have achieved strong performance on vision-language tasks, particularly Visual Question Answering (VQA). While prior work has explored unimodal biases in VQA, the problem of selection…
Visual Question AnsweringSemantic SimilarityVisual Reasoning