FiVL: A Framework for Improved Vision-Language Alignment
Large Vision Language Models (LVLMs) have achieved significant progress in integrating visual and textual inputs for multimodal reasoning. However, a recurring challenge is ensuring these models utilize visual information as effectively as linguistic content when both modalities are necessary to formulate an accurate answer. We hypothesize that hallucinations arise due to the lack of effective visual grounding in current LVLMs. This issue extends to vision-language benchmarks, where it is difficult to make the image indispensable for accurate answer generation, particularly in vision question-answering tasks. In this work, we introduce FiVL, a novel method for constructing datasets designed to train LVLMs for enhanced visual grounding and to evaluate their effectiveness in achieving it. These datasets can be utilized for both training and assessing an LVLM's ability to use image content as substantive evidence rather than relying solely on linguistic priors, providing insights into the model's reliance on visual information. To demonstrate the utility of our dataset, we introduce an innovative training task that outperforms baselines alongside a validation method and application for explainability. The code is available at https://github.com/IntelLabs/fivl.
Code (0)
등록된 구현이 없습니다.
Tasks
Answer GenerationMultimodal ReasoningQuestion AnsweringVisual GroundingSimilar Papers 제목 키워드 기반
GTA-Net: Cooperative Game Theory for Vision-Language Alignment in Chest X-Ray Report Generation
Automated chest X-ray report generation requires precise cross-modal grounding to ensure clinically reliable descriptions. However, existing vision-language models rely on implicit attention mechanisms that fail to enfor…
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs o…
A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene
Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focus…
image-classificationImage ClassificationKnowledge DistillationLanguage Modelling+1Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study
Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. Since mBERT is not pre-trained with explicit cross-lingual supervision, transfer perform…
Cross-Lingual Transferslot-fillingSlot FillingZero-Shot Cross-Lingual TransferAlign Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment
The widespread deployment of large language models (LLMs) across linguistic communities necessitates reliable multilingual safety alignment. However, recent efforts to extend alignment to other languages often require su…