Reasoning is a Modality

2026-01-20 · Zhiguang Liu, Yi Shang arxiv

The Abstraction and Reasoning Corpus (ARC) provides a compact laboratory for studying abstract reasoning, an ability central to human intelligence. Modern AI systems, including LLMs and ViTs, largely operate as sequence-of-behavior prediction machines: they match observable behaviors by modeling token statistics without a persistent, readable mental state. This creates a gap with human-like behavior: humans can explain an action by decoding internal state, while AI systems can produce fluent post-hoc rationalizations that are not grounded in such a state. We hypothesize that reasoning is a modality: reasoning should exist as a distinct channel separate from the low-level workspace on which rules are applied. To test this hypothesis, on solving ARC tasks as a visual reasoning problem, we designed a novel role-separated transformer block that splits global controller tokens from grid workspace tokens, enabling iterative rule execution. Trained and evaluated within the VARC vision-centric protocol, our method achieved 62.6% accuracy on ARC-1, surpassing average human performance (60.2%) and outperforming prior methods significantly. Qualitatively, our models exhibit more coherent rule-application structure than the dense ViT baseline, consistent with a shift away from plausible probability blobs toward controller-driven reasoning.

📄 PDF Abstract BibTeX arXiv:2601.13562

Code (0)

등록된 구현이 없습니다.

Tasks

Visual Reasoning

Learning What to Attend First: Modality-Importance-Guided Reasoning for Reliable Multimodal Emotion Understanding

2025-12-02 · Hyeongseop Rha, Jeong Hun Yeo, Junil Won, Se Jin Park 외 arxiv

In this paper, we present Modality-Importance-Guided Reasoning (MIGR), a framework designed to improve the reliability of reasoning-based multimodal emotion understanding in multimodal large language models. Although exi…

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

2026-05-11 · Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang 외 arxiv

Audio and vision provide complementary evidence for audio-visual question answering, yet current audio-visual large language models may suffer from cross-modal interference: information from one modality misguides the in…

Audio-visual Question Answering Reinforcement Learning Visual Reasoning

X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis

2026-04-22 · Gui Wang, Zehao Zhong, YongSong Zhou, Yudong Li 외 arxiv

Despite significant progress in Multi-modal Large Language Models (MLLMs), their clinical reasoning capacity for multi-modal diagnosis remains largely unexamined. Current benchmarks, mostly single-modality data, can't ev…

Image Quality Assessment

Spatial Reasoning via Modality Switching Between Language and Symbolic Representation

2026-06-30 · Shreya Rajpal, Tanawan Premsri, Parisa Kordjamshidi arxiv

Human reasoning is inherently multimodal: when problems become difficult, we rarely think in words alone. We often externalize our reasoning by sketching diagrams or drawing grids to understand the underlying conceptual …

Spatial Reasoning

RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding

2026-01-31 · Jiahe Wu, Bing Cao, Qilong Wang, Qinghua Hu 외 arxiv

Multimodal Large Language Models (MLLM) are primarily pre-trained on the RGB modality, thereby limiting their performance on other modalities, such as infrared, depth, and event data, which are crucial for complex scenar…

Reinforcement Learning