Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias
Common methods for interpreting neural models in natural language processing typically examine either their structure or their behavior, but not both. We propose a methodology grounded in the theory of causal mediation analysis for interpreting which parts of a model are causally implicated in its behavior. It enables us to analyze the mechanisms by which information flows from input to output through various model components, known as mediators. We apply this methodology to analyze gender bias in pre-trained Transformer language models. We study the role of individual neurons and attention heads in mediating gender bias across three datasets designed to gauge a model's sensitivity to gender bias. Our mediation analysis reveals that gender bias effects are (i) sparse, concentrated in a small part of the network; (ii) synergistic, amplified or repressed by different components; and (iii) decomposable into effects flowing directly from the input and indirectly through the mediators.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Many interpretation methods for neural models in natural language processing investigate how information is encoded inside hidden representations. However, these methods can only measure whether the information exists, n…
Interpreting Bias in Large Language Models: A Feature-Based Approach
Large Language Models (LLMs) such as Mistral and LLaMA have showcased remarkable performance across various natural language processing (NLP) tasks. Despite their success, these models inherit social biases from the dive…
counterfactualWhat Changed? Investigating Debiasing Methods using Causal Mediation Analysis
Previous work has examined how debiasing language models affect downstream tasks, specifically, how debiasing techniques influence task performance and whether debiased models also make impartial predictions in downstrea…
Scaling Causal Mediation for Complex Systems: A Framework for Root Cause Analysis
Modern operational systems ranging from logistics and cloud infrastructure to industrial IoT, are governed by complex, interdependent processes. Understanding how interventions propagate through such systems requires cau…
Causal InferenceTotal-effect Test May Erroneously Reject So-called "Full" or "Complete" Mediation
The procedure for establishing mediation, i.e., determining that an independent variable X affects a dependent variable Y through some mediator M, has been under debate. The classic causal steps require that a "total eff…
Mathematical Proofs