Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains

2026-01-29 · Yuqi Xiong, Chunyi Peng, Zhipeng Xu, Zhenghao Liu, Zulong Chen, Yukun Yan, Shuo Wang, Yu Gu, Ge Yu arxiv

Visual Retrieval-Augmented Generation (VRAG) enhances Vision-Language Models (VLMs) by incorporating external visual documents to address a given query. Existing VRAG frameworks usually depend on rigid, pre-defined external tools to extend the perceptual capabilities of VLMs, typically by explicitly separating visual perception from subsequent reasoning processes. However, this decoupled design can lead to unnecessary loss of visual information, particularly when image-based operations such as cropping are applied. In this paper, we propose Lang2Act, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains. Rather than invoking fixed external engines, Lang2Act collects self-emergent actions as linguistic tools and leverages them to enhance the visual perception capabilities of VLMs. To support this mechanism, we design a two-stage Reinforcement Learning (RL)-based training framework. Specifically, the first stage optimizes VLMs to self-explore high-quality actions for constructing a reusable linguistic toolbox, and the second stage further optimizes VLMs to exploit these linguistic tools for downstream reasoning effectively. Experimental results demonstrate the effectiveness of Lang2Act in substantially enhancing the visual perception capabilities of VLMs, achieving performance improvements of over 4%. All code and data are available at https://github.com/NEUIR/Lang2Act.

📄 PDF Abstract BibTeX arXiv:2602.13235

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Visual Reasoning

FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision-Language Models

2026-01-29 · Chenyu Huang, Peng Ye, Xudong Tan, Jinhan Mu 외 arxiv

Efficiently enhancing the reasoning capabilities of Vision-Language Models (VLMs) by merging them with Large Reasoning Models (LRMs) has emerged as a promising direction. However, existing methods typically operate at a …

Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models

2026-02-19 · Dhruba Ghosh, Yuhui Zhang, Ludwig Schmidt arxiv

Vision-language models (VLMs) have made substantial progress across a wide range of visual question answering benchmarks, spanning visual reasoning, document understanding, and multimodal dialogue. These improvements are…

Visual Question Answering Image Classification Visual Reasoning

RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning

2025-10-02 · Sicheng Feng, Kaiwen Tuo, Song Wang, Lingdong Kong 외 arxiv

Fine-grained visual reasoning remains a core challenge for multimodal large language models (MLLMs). The recently introduced ReasonMap highlights this gap by showing that even advanced MLLMs struggle with spatial reasoni…

Visual Question Answering Reinforcement Learning Spatial Reasoning Visual Reasoning

ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling

2024-02-09 · Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou 외

By combining natural language understanding, generation capabilities, and breadth of knowledge of large language models with image perception, recent large vision language models (LVLMs) have shown unprecedented visual r…

Hallucination Natural Language Understanding Visual Grounding Visual Reasoning

Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding

2025-08-23 · Leilei Guo, Antonio Carlos Rivera, Peiyu Tang, Haoxuan Ren 외 arxiv

Large Language Models (LLMs) and Vision-Language Large Models (LVLMs) have achieved remarkable progress in natural language processing and multimodal understanding. Despite their impressive generalization capabilities, c…

Referring Expression Visual Reasoning