On Mechanistic Circuits for Extractive Question-Answering
Large language models are increasingly used to process documents and facilitate question-answering on them. In our paper, we extract mechanistic circuits for this real-world language modeling task: context-augmented language modeling for extractive question-answering (QA) tasks and understand the potential benefits of circuits towards downstream applications such as data attribution to context information. We extract circuits as a function of internal model components (e.g., attention heads, MLPs) using causal mediation analysis techniques. Leveraging the extracted circuits, we first understand the interplay between the model's usage of parametric memory and retrieved context towards a better mechanistic understanding of context-augmented language models. We then identify a small set of attention heads in our circuit which performs reliable data attribution by default, thereby obtaining attribution for free in just the model's forward pass. Using this insight, we then introduce ATTNATTRIB, a fast data attribution algorithm which obtains state-of-the-art attribution results across various extractive QA benchmarks. Finally, we show the possibility to steer the language model towards answering from the context, instead of the parametric memory by using the attribution from ATTNATTRIB as an additional signal during the forward pass. Beyond mechanistic understanding, our paper provides tangible applications of circuits in the form of reliable data attribution and model steering.
Code (0)
등록된 구현이 없습니다.
Tasks
Extractive Question-AnsweringLanguage ModelingLanguage ModellingQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MemSum-DQA: Adapting An Efficient Long Document Extractive Summarizer for Document Question Answering
We introduce MemSum-DQA, an efficient system for document question answering (DQA) that leverages MemSum, a long document extractive summarizer. By prefixing each text block in the parsed document with the provided quest…
Extractive SummarizationQuestion AnsweringAttention-guided Generative Models for Extractive Question Answering
We propose a novel method for applying Transformer models to extractive question answering (QA) tasks. Recently, pretrained generative sequence-to-sequence (seq2seq) models have achieved great success in question answeri…
DecoderExtractive Question-AnsweringHallucinationOpen-Domain Question Answering+2GSQA: An End-to-End Model for Generative Spoken Question Answering
In recent advancements in spoken question answering (QA), end-to-end models have made significant strides. However, previous research has primarily focused on extractive span selection. While this extractive-based approa…
Question AnsweringMechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
Fine-tuning large language models (LLMs) frequently induces catastrophic forgetting of prior capabilities. Recent work has shown that reinforcement learning (RL) retains prior capabilities more effectively than supervise…
Reinforcement LearningTowards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP
Mechanistic interpretability seeks to understand the neural mechanisms that enable specific behaviors in Large Language Models (LLMs) by leveraging causality-based methods. While these approaches have identified neural c…
Question AnsweringText GenerationVisual Question Answering