Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification

2026-01-23 · Zongwan Cao, Bingbing Wen, Lucy Lu Wang arxiv

Real-world visual question answering (VQA) is often context-dependent: an image-question pair may be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose CoA(Clarify-or-Answer), an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it asks a single focused question and then incorporates the response to produce the final answer. We introduce CONTEXTCLARIFY with a set of ambiguous VQA questions and the contrast set that is non-ambiguous. We further introduce GRPO-CR (Clarification Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLLMs and three datasets, CoA achieves consistent improvements at both the module and system levels, improving end-to-end VQA accuracy by an average of +15.3 points (83%) over prompting-based baselines

📄 PDF Abstract BibTeX arXiv:2601.16400

Code (0)

등록된 구현이 없습니다.

Tasks

Visual Question Answering Reinforcement Learning Question Generation

ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models

2025-12-24 · Sichun Luo, Yi Huang, Mukai Li, Shichang Meng 외 arxiv

Large language models (LLMs) are increasingly deployed as conversational assistants in open-domain, multi-turn settings, where users often provide incomplete or ambiguous information. However, existing LLM-focused clarif…

Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems

2026-04-01 · Dongzhe Fan, Zheyi Xue, Siyuan Liu, Qiaoyu Tan arxiv

Retrieval-augmented generation (RAG) and its graph-based extensions (GraphRAG) are effective paradigms for improving large language model (LLM) reasoning by grounding generation in external knowledge. However, most exist…

Question Answering

Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions

2026-05-24 · Jinyan Su, Claire Cardie arxiv

User queries are often underspecified and may admit multiple valid interpretations. Rather than silently making assumptions about the user's intent, a helpful assistant should surface such ambiguity by asking a clarifyin…

Question Answering

Controlling the Risk of Conversational Search via Reinforcement Learning

2021-01-15 · Zhenduo Wang, Qingyao Ai

Users often formulate their search queries with immature language without well-developed keywords and complete structures. Such queries fail to express their true information needs and raise ambiguity as fragmental langu…

Conversational Search reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Learning to Clarify by Reinforcement Learning Through Reward-Weighted Fine-Tuning

2025-06-08 · Subhojyoti Mukherjee, Viet Dac Lai, Raghavendra Addanki, Ryan Rossi 외

Question answering (QA) agents automatically answer questions posed in natural language. In this work, we learn to ask clarifying questions in QA agents. The key idea in our method is to simulate conversations that conta…

Offline RL Question Answering Reinforcement Learning (RL)