AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs
Generating diverse and effective clarifying questions is crucial for improving query understanding and retrieval performance in open-domain conversational search (CS) systems. We propose AGENT-CQ (Automatic GENeration, and evaluaTion of Clarifying Questions), an end-to-end LLM-based framework addressing the challenges of scalability and adaptability faced by existing methods that rely on manual curation or template-based approaches. AGENT-CQ consists of two stages: a generation stage employing LLM prompting strategies to generate clarifying questions, and an evaluation stage (CrowdLLM) that simulates human crowdsourcing judgments using multiple LLM instances to assess generated questions and answers based on comprehensive quality metrics. Extensive experiments on the ClariQ dataset demonstrate CrowdLLM's effectiveness in evaluating question and answer quality. Human evaluation and CrowdLLM show that the AGENT-CQ - generation stage, consistently outperforms baselines in various aspects of question and answer quality. In retrieval-based evaluation, LLM-generated questions significantly enhance retrieval effectiveness for both BM25 and cross-encoder models compared to human-generated questions.
Code (0)
등록된 구현이 없습니다.
Tasks
Conversational SearchRetrievalSimilar Papers 제목 키워드 기반
Analyzing and Learning from User Interactions for Search Clarification
Asking clarifying questions in response to search queries has been recognized as a useful technique for revealing the underlying intent of the query. Clarification has applications in retrieval systems with different int…
Re-RankingRetrievalEvaluating Mixed-initiative Conversational Search Systems via User Simulation
Clarifying the underlying user information need by asking clarifying questions is an important feature of modern conversational search system. However, evaluation of such systems through answering prompted clarifying que…
Conversational SearchText GenerationUser SimulationPseudo Ambiguous and Clarifying Questions Based on Sentence Structures Toward Clarifying Question Answering System
Question answering (QA) with disambiguation questions is essential for practical QA systems because user questions often do not contain information enough to find their answers. We call this task clarifying question answ…
Question AnsweringQuestion GenerationQuestion-GenerationSentenceLearning to Clarify by Reinforcement Learning Through Reward-Weighted Fine-Tuning
Question answering (QA) agents automatically answer questions posed in natural language. In this work, we learn to ask clarifying questions in QA agents. The key idea in our method is to simulate conversations that conta…
Offline RLQuestion AnsweringReinforcement Learning (RL)Deciding Whether to Ask Clarifying Questions in Large-Scale Spoken Language Understanding
A large-scale conversational agent can suffer from understanding user utterances with various ambiguities such as ASR ambiguity, intent ambiguity, and hypothesis ambiguity. When ambiguities are detected, the agent should…
Spoken Language Understanding