Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding
Conversational query clarification enables users to refine their search queries through interactive dialogue, improving search effectiveness. Traditional approaches rely on text-based clarifying questions, which often fail to capture complex user preferences, particularly those involving visual attributes. While recent work has explored single-turn multi-modal clarification with images alongside text, such methods do not fully support the progressive nature of user intent refinement over multiple turns. Motivated by this, we introduce the Multi-turn Multi-modal Clarifying Questions (MMCQ) task, which combines text and visual modalities to refine user queries in a multi-turn conversation. To facilitate this task, we create a large-scale dataset named ClariMM comprising over 13k multi-turn interactions and 33k question-answer pairs containing multi-modal clarifying questions. We propose Mario, a retrieval framework that employs a two-phase ranking strategy: initial retrieval with BM25, followed by a multi-modal generative re-ranking model that integrates textual and visual information from conversational history. Our experiments show that multi-turn multi-modal clarification outperforms uni-modal and single-turn approaches, improving MRR by 12.88%. The gains are most significant in longer interactions, demonstrating the value of progressive refinement for complex queries.
Code (0)
등록된 구현이 없습니다.
Tasks
Re-RankingRetrievalSimilar Papers 제목 키워드 기반
Clarification Is Not Enough: Post-Clarification Answering Remains the Bottleneck in Multi-Turn QA
Pluralistic alignment requires systems to adapt to diverse user values, communication styles, and contextual assumptions. We believe that a foundational prerequisite for such alignment enabling accurate preference elicit…
Question AnsweringOne More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies
Ambiguous user requests make clarification a sequential decision problem for conversational LLM assistants: they must decide whether to ask, what to ask, when to stop, and when to answer. We introduce RegretBench, a mult…
Product RecommendationCLAM: Selective Clarification for Ambiguous Questions with Generative Language Models
Users often ask dialogue systems ambiguous questions that require clarification. We show that current language models rarely ask users to clarify ambiguous questions and instead provide incorrect answers. To address this…
Language ModellingQuestion AnsweringTriviaQASelf-supervised clarification question generation for ambiguous multi-turn conversation
Clarification Question Generation (CQG) aims to automatically generate clarification questions to avoid misunderstanding. In this paper, we focus on generating clarification questions in the scenario of ambiguous multi-t…
Conversational RecommendationQuestion GenerationQuestion-GenerationRecommendation Systems+1Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search
In mixed-initiative conversational search systems, clarifying questions are used to help users who struggle to express their intentions in a single query. These questions aim to uncover user's information needs and resol…
4kConversational Search