Counting to Four is still a Chore for VLMs

2026-04-11 · Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo arxiv

Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

📄 PDF Abstract BibTeX arXiv:2604.10039

Code (0)

등록된 구현이 없습니다.

Tasks

Multimodal Reasoning Answer Generation Object Counting

The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs

2026-07-10 · Ahmed Oumar El-Shangiti, Abzal Nurgazy, Hilal AlQuabeh, Nikolai Rozanov 외 arxiv

Despite strong performance on many multimodal tasks, vision-language models (VLMs) still struggle with basic object counting. We investigate whether this reflects missing internal knowledge or a gap between internal repr…

Object Counting

Vision Language Models are Biased

2025-05-29 · An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang 외

Large language models (LLMs) memorize a vast amount of prior knowledge from the Internet that help them on downstream tasks but also may notoriously sway their outputs towards wrong or biased answers. In this work, we te…

Board Games counterfactual

Assessing the Visual Enumeration Abilities of Specialized Counting Architectures and Vision-Language Models

2025-12-17 · Kuinan Hou, Jing Mi, Marco Zorzi, Lamberto Ballan 외 arxiv

Counting the number of items in a visual scene remains a fundamental yet challenging task in computer vision. Traditional approaches to solving this problem rely on domain-specific counting architectures, which are train…

Object Counting

[De|Re]constructing VLMs' Reasoning in Counting

2025-10-22 · Simone Alghisi, Gabriel Roccabruna, Massimo Rizzoli, Seyed Mahed Mousavi 외 arxiv

Vision-Language Models (VLMs) have recently gained attention due to their competitive performance on multiple downstream tasks, achieved by following user-input instructions. However, VLMs still exhibit several limitatio…

Temporal Sequences Visual Reasoning

GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations

2026-03-11 · Boyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri 외 arxiv

Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-…

Object Detection Visual Reasoning