Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models
Visual corruptions can change vision--language model (VLM) behavior in ways that top-1 accuracy does not capture. A model may keep the same answer while losing distributional support, or improve accuracy through unstable wrong-to-correct changes. We introduce Bench-C, a controlled multiple-choice testbed for studying these effects. It selects semantically diverse samples whose predictions respond to corruption, and evaluates them under 19 corruption types and five severity levels. To measure how corruption changes the option distribution, we introduce the Robustness Alignment Score (RAS), which combines confidence-correctness alignment with uncertainty direction. We further separate originally correct samples from originally wrong samples, and track whether changes are temporary or persistent across severity. Experiments across 13 VLMs reveal a counterintuitive pattern: mild corruptions can improve top-1 accuracy while degrading prediction structure. These failures include silent degradation, erroneous overconfidence, and severity-dependent persistence. Bench-C therefore supports robustness evaluation that goes beyond final answers and attributes where reliability changes occur. Code and data are available at https://github.com/xiangjieSui/Bench-C.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Exploring Silent Data Corruption as a Reliability Challenge in LLM Training
As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults t…
LLM-Based Automated Diagnosis Of Integration Test Failures At Google
Integration testing is critical for the quality and reliability of complex software systems. However, diagnosing their failures presents significant challenges due to the massive volume, unstructured nature, and heteroge…
INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs
Despite rapid progress, Video Large Language Models (Video-LLMs) remain unreliable due to hallucinations, which are outputs that contradict either video evidence (faithfulness) or verifiable world knowledge (factuality).…
NICE: Scale-Stable Perturbations for Graph Neural Network Explanations via Noise Corruption
Post-hoc Graph Neural Network (GNN) explainers commonly follow a Perturb-Query paradigm, inferring the importance of graph elements based on queried predictions to perturbed inputs. However, such perturbations often intr…
Graph Neural NetworkDEBUG-HD: Debugging TinyML models on-device using Hyper-Dimensional computing
TinyML models often operate in remote, dynamic environments without cloud connectivity, making them prone to failures. Ensuring reliability in such scenarios requires not only detecting model failures but also identifyin…