Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents
Instructional documents are rich sources of knowledge for completing various tasks, yet their unique challenges in conversational question answering (CQA) have not been thoroughly explored. Existing benchmarks have primarily focused on basic factual question-answering from single narrative documents, making them inadequate for assessing a model`s ability to comprehend complex real-world instructional documents and provide accurate step-by-step guidance in daily life. To bridge this gap, we present InsCoQA, a novel benchmark tailored for evaluating large language models (LLMs) in the context of CQA with instructional documents. Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents, reflecting the intricate and multi-faceted nature of real-world instructional tasks. Additionally, to comprehensively assess state-of-the-art LLMs on the InsCoQA benchmark, we propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingConversational Question AnsweringQuestion AnsweringSimilar Papers 제목 키워드 기반
MMCoQA: Conversational Question Answering over Text, Tables, and Images
The rapid development of conversational assistants accelerates the study on conversational question answering (QA). However, the existing conversational QA systems usually answer users’ questions with a single knowledge …
BenchmarkingConversational Question AnsweringQuestion AnsweringRetrievalAutomated Factual Benchmarking for In-Car Conversational Systems using Large Language Models
In-car conversational systems bring the promise to improve the in-vehicle user experience. Modern conversational systems are based on Large Language Models (LLMs), which makes them prone to errors such as hallucinations,…
BenchmarkingConversational Question AnsweringQuestion AnsweringDisambiguation in Conversational Question Answering in the Era of LLM: A Survey
Ambiguity remains a fundamental challenge in Natural Language Processing (NLP) due to the inherent complexity and flexibility of human language. With the advent of Large Language Models (LLMs), addressing ambiguity has b…
BenchmarkingConversational Question AnsweringQuestion AnsweringPerkwE_COQA: Enhanced Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models
Smart cities need the involvement of their residents to enhance quality of life. Conversational query-answering is an emerging approach for user engagement. There is an increasing demand of an advanced conversational que…
Conversational Question AnsweringKeyword ExtractionQuestion AnsweringSpeeding Up Question Answering Task of Language Models via Inverted Index
Natural language processing applications, such as conversational agents and their question-answering capabilities, are widely used in the real world. Despite the wide popularity of large language models (LLMs), few real-…
Question Answering