Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networks
Interpretability research takes counterfactual theories of causality for granted. Most causal methods rely on counterfactual interventions to inputs or the activations of particular model components, followed by observations of the change in models' output logits or behaviors. While this yields more faithful evidence than correlational methods, counterfactuals nonetheless have key problems that bias our findings in specific and predictable ways. Specifically, (i) counterfactual theories do not effectively capture multiple independently sufficient causes of the same effect, which leads us to miss certain causes entirely; and (ii) counterfactual dependencies in neural networks are generally not transitive, which complicates methods for extracting and interpreting causal graphs from neural networks. We discuss the implications of these challenges for interpretability researchers and propose concrete suggestions for future work.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Inferring the size of the causal universe: features and fusion of causal attribution networks
Cause-and-effect reasoning, the attribution of effects to causes, is one of the most powerful and unique skills humans possess. Multiple surveys are mapping out causal attributions as networks, but it is unclear how well…
Non-classicality in mental states : an experimental study with ambiguous audio (music) stimuli
This paper attempts to address the question that whether the present physical or mathematical theories are sufficient to understand the complexities of human brain when it interacts with the external environment in the f…
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbati…
counterfactualText GenerationMulti-objective Explanations of GNN Predictions
Graph Neural Network (GNN) has achieved state-of-the-art performance in various high-stake prediction tasks, but multiple layers of aggregations on graphs with irregular structures make GNN a less interpretable model. Pr…
counterfactualDecision MakingFairnessGraph Neural NetworkCauses and Explanations: A Structural-Model Approach --- Part 1: Causes
We propose a new definition of actual causes, using structural equations to model counterfactuals.We show that the definitions yield a plausible and elegant account ofcausation that handles well examples which have cause…