Pathological Truth Bias in Vision-Language Models
Vision Language Models (VLMs) are improving quickly, but standard benchmarks can hide systematic failures that reduce real world trust. We introduce MATS (Multimodal Audit for Truthful Spatialization), a compact behavioral audit that measures whether models reject visually contradicted statements, and two metrics Spatial Consistency Score (SCS) and Incorrect Agreement Rate (IAR). Instruction tuned generative VLMs (LLaVA 1.5, QwenVLchat) exhibit very low SCS and high IAR, while contrastive encoders (CLIP, SigLIP) are far more robust. Activation patching causally localizes failure loci (mid to late cross attention for generative models, pooled projection components for contrastive models) and suggests concrete repair paths.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
The LIDC-IDRI database is the most popular benchmark for lung cancer prediction. However, with subjective assessment from radiologists, nodules in LIDC may have entirely different malignancy annotations from the patholog…
Metric LearningPredictionRetrievalShape-aware synthesis of pathological lung CT scans using CycleGAN for enhanced semi-supervised lung segmentation
This paper addresses the problem of pathological lung segmentation, a significant challenge in medical image analysis, particularly pronounced in cases of peripheral opacities (severe fibrosis and consolidation) because …
Data AugmentationImage SegmentationImage-to-Image TranslationMedical Image Analysis+3CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images
Purpose: This study aimed to develop an open-source multimodal large language model (CXR-LLAVA) for interpreting chest X-ray images (CXRs), leveraging recent advances in large language models (LLMs) to potentially replic…
DiagnosticLanguage ModelingLanguage ModellingLarge Language Model+3LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation
Radiology Report Generation (RRG) aims to produce accurate and coherent diagnostics from medical images. Although large vision language models (LVLM) improve report fluency and accuracy, they exhibit hallucinations, gene…
Libra-MIL: Multimodal Prototypes Stereoscopic Infused with Task-specific Language Priors for Few-shot Whole Slide Image Classification
While Large Language Models (LLMs) are emerging as a promising direction in computational pathology, the substantial computational cost of giga-pixel Whole Slide Images (WSIs) necessitates the use of Multi-Instance Learn…
Image Classification