Evaluating Large Language Models for Document-grounded Response Generation in Information-Seeking Dialogues
In this paper, we investigate the use of large language models (LLMs) like ChatGPT for document-grounded response generation in the context of information-seeking dialogues. For evaluation, we use the MultiDoc2Dial corpus of task-oriented dialogues in four social service domains previously used in the DialDoc 2022 Shared Task. Information-seeking dialogue turns are grounded in multiple documents providing relevant information. We generate dialogue completion responses by prompting a ChatGPT model, using two methods: Chat-Completion and LlamaIndex. ChatCompletion uses knowledge from ChatGPT model pretraining while LlamaIndex also extracts relevant information from documents. Observing that document-grounded response generation via LLMs cannot be adequately assessed by automatic evaluation metrics as they are significantly more verbose, we perform a human evaluation where annotators rate the output of the shared task winning system, the two Chat-GPT variants outputs, and human responses. While both ChatGPT variants are more likely to include information not present in the relevant segments, possibly including a presence of hallucinations, they are rated higher than both the shared task winning system and human responses.
Code (0)
등록된 구현이 없습니다.
Tasks
Response GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs
A major concern in using deep learning based generative models for document-grounded dialogs is the potential generation of responses that are not \textit{faithful} to the underlying document. Existing automated metrics …
Response GenerationEvaluating Generative Ad Hoc Information Retrieval
Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded gener…
Ad-Hoc Information RetrievalDocument RankingInformation RetrievalRetrievalSelf-Refinement of Language Models from External Proxy Metrics Feedback
It is often desirable for Large Language Models (LLMs) to capture multiple objectives when providing a response. In document-grounded response generation, for example, agent responses are expected to be relevant to a use…
Question AnsweringResponse GenerationDesigning an Evaluation Framework for Large Language Models in Astronomy Research
Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communities like astronomy might benefit from th…
AstronomyChatbotExperimental DesignRAG+2A Pre-training Strategy for Zero-Resource Response Selection in Knowledge-Grounded Conversations
Recently, many studies are emerging towards building a retrieval-based dialogue system that is able to effectively leverage background knowledge (e.g., documents) when conversing with humans. However, it is non-trivial t…
Language ModellingRetrievalTask 2