Retrieving Versus Understanding Extractive Evidence in Few-Shot Learning
A key aspect of alignment is the proper use of within-document evidence to construct document-level decisions. We analyze the relationship between the retrieval and interpretation of within-document evidence for large language model in a few-shot setting. Specifically, we measure the extent to which model prediction errors are associated with evidence retrieval errors with respect to gold-standard human-annotated extractive evidence for five datasets, using two popular closed proprietary models. We perform two ablation studies to investigate when both label prediction and evidence retrieval errors can be attributed to qualities of the relevant evidence. We find that there is a strong empirical relationship between model prediction and evidence retrieval error, but that evidence retrieval error is mostly not associated with evidence interpretation error--a hopeful sign for downstream applications built on this mechanism.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot LearningLanguage ModelingLanguage ModellingLarge Language ModelPredictionRetrievalSimilar Papers 제목 키워드 기반
Extractive Summarization: Limits, Compression, Generalized Model and Heuristics
Due to its promise to alleviate information overload, text summarization has attracted the attention of many researchers. However, it has remained a serious challenge. Here, we first prove empirical limits on the recall …
Document SummarizationExtractive SummarizationmodelMulti-Document Summarization+1Exploring Language Model Generalization in Low-Resource Extractive QA
In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize to domains that require specific knowledge such as medicine and law in a z…
Domain GeneralizationExtractive Question-AnsweringLanguage ModelingLanguage Modelling+1N2N-GQA: Noise-to-Narrative for Graph-Based Table-Text Question Answering Using LLMs
Multi-hop question answering over hybrid table-text data requires retrieving and reasoning across multiple evidence pieces from large corpora, but standard Retrieval-Augmented Generation (RAG) pipelines process documents…
Multi-hop Question AnsweringEASE: Extractive-Abstractive Summarization End-to-End using the Information Bottleneck Principle
Current abstractive summarization systems outperform their extractive counterparts, but their widespread adoption is inhibited by the inherent lack of interpretability. Extractive summarization systems, though interpreta…
Abstractive Text SummarizationExtractive SummarizationText GenerationUnderstanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering
Generative question answering (QA) models generate answers to questions either solely based on the parameters of the model (the closed-book setting) or additionally retrieving relevant evidence (the open-book setting). G…
Generative Question AnsweringQuestion Answering