Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction
Large language models require updates to remain up-to-date or adapt to new domains by fine-tuning them with new documents. One key is memorizing the latest information in a way that the memorized information is extractable with a query prompt. However, LLMs suffer from a phenomenon called perplexity curse; despite minimizing document perplexity during fine-tuning, LLMs struggle to extract information through a prompt sentence. In this new knowledge acquisition and extraction, we find a very intriguing fact that LLMs can accurately answer questions about the first sentence, but they struggle to extract information described in the middle or end of the documents used for fine-tuning. Our study suggests that the auto-regressive training causes this issue; each token is prompted by reliance on all previous tokens, which hinders the model from recalling information from training documents by question prompts. To conduct the in-depth study, we publish both synthetic and real datasets, enabling the evaluation of the QA performance w.r.t. the position of the corresponding answer in a document. Our investigation shows that even a large model suffers from the perplexity curse, but regularization such as denoising auto-regressive loss can enhance the information extraction from diverse positions. These findings will be (i) a key to improving knowledge extraction from LLMs and (ii) new elements to discuss the trade-off between RAG and fine-tuning in adapting LLMs to a new domain.
Code (1)
Tasks
DenoisingLanguage ModelingLanguage ModellingQuestion AnsweringRAGSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On Evaluating Multilingual Compositional Generalization with Translated Datasets
Compositional generalization allows efficient learning and human-like inductive biases. Since most research investigating compositional generalization in NLP is done on English, important questions remain underexplored. …
Machine TranslationSemantic ParsingTranslationDo Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation
Large language models (LLMs) are increasingly used to generate multiple-choice questions (MCQs), where correct answers should ideally be uniformly distributed across options. However, we observe that LLMs exhibit systema…
Question GenerationVerbosity Bias in Preference Labeling by Large Language Models
In recent years, Large Language Models (LLMs) have witnessed a remarkable surge in prevalence, altering the landscape of natural language processing and machine learning. One key factor in improving the performance of LL…
reinforcement-learningReinforcement LearningA Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks
We investigate 17 benchmarks (e.g. SugarCREPE, VALSE) commonly used for measuring compositional understanding capabilities of vision-language models (VLMs). We scrutinize design choices in their construction, including d…
Language ModelingLanguage ModellingExploiting Primacy Effect To Improve Large Language Models
Large Language Models (LLMs) have become essential in many Natural Language Processing (NLP) tasks, leveraging extensive pre-training and fine-tuning to achieve high accuracy. However, like humans, LLMs exhibit biases, p…
Semantic SimilarityQuestion Answering