Spatially Grounded Concept-Based Image Classification
Deep neural networks can achieve high accuracy while relying on evidence that is hard to inspect or misaligned with the intended task. Concept Bottleneck Models (CBMs) expose human-interpretable concepts, but most treat concepts as global attributes and do not show how localized evidence is aggregated into a decision. We propose \textbf{SEG-MIL-CBM}, a spatially grounded CBM that decomposes each image into concept-guided regions and classifies it by attention-based aggregation of segment-level concept evidence. The same segment evidence terms form the prediction and the explanation, exposing which regions and concepts support the predicted logit without a separate post-hoc attribution module. Among evaluated CBM-family baselines, SEG-MIL-CBM improves Waterbirds worst-group accuracy from $65.1\%$ to $72.0\%$, reaches $87.4\%$ worst-group accuracy on Pawrious, remains competitive on standard recognition, and attains the best CBM accuracy on CIFAR-100 ($85.3\%$). Segment-level faithfulness experiments on CUB further show that its learned segment ranking matches or improves over evaluated segment-ranking controls.
Code (0)
등록된 구현이 없습니다.
Tasks
Image ClassificationResults from the Paper
| Rank | Task | Dataset | Model | Metrics |
|---|---|---|---|---|
| #2 | Classification | CIFAR-100 | Spatially Grounded Concept-Based Image C | Accuracy: 85.3 |
| #65 | Image Classification | CIFAR-100 | Spatially Grounded Concept-Based Image C | Percentage correct: 85.3 |
Similar Papers 제목 키워드 기반
CFM: Language-aligned Concept Foundation Model for Vision
Language-aligned vision foundation models perform strongly across diverse downstream tasks. Yet, their learned representations remain opaque, making interpreting their decision-making difficult. Recent work decompose the…
Image ClassificationShow and Tell: Visually Explainable Deep Neural Nets via Spatially-Aware Concept Bottleneck Models
Modern deep neural networks have now reached human-level performance across a variety of tasks. However, unlike humans they lack the ability to explain their decisions by showing where and telling what concepts guided th…
Zero Shot SegmentationBeyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI
Spatial reasoning and visual grounding are core capabilities for vision-language models (VLMs), yet most medical VLMs produce predictions without transparent reasoning or spatial evidence. Existing benchmarks also evalua…
Visual Question AnsweringSpatial ReasoningVisual GroundingSpatially Grounded Long-Horizon Task Planning in the Wild
Recent advances in robot manipulation increasingly leverage Vision-Language Models (VLMs) for high-level reasoning, such as decomposing task instructions into sequential action plans expressed in natural language that gu…
Robot ManipulationGH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks
Systematic failures of vision models on semantically coherent subsets, known as error slices, reveal limitations in robustness and evaluation. Existing slice discovery approaches largely model slices as clusters in repre…
Object Detection