AttributionLab: Faithfulness of Feature Attribution Under Controllable Environments
Feature attribution explains neural network outputs by identifying relevant input features. The attribution has to be faithful, meaning that the attributed features must mirror the input features that influence the output. One recent trend to test faithfulness is to fit a model on designed data with known relevant features and then compare attributions with ground truth input features.This idea assumes that the model learns to use all and only these designed features, for which there is no guarantee. In this paper, we solve this issue by designing the network and manually setting its weights, along with designing data. The setup, AttributionLab, serves as a sanity check for faithfulness: If an attribution method is not faithful in a controlled environment, it can be unreliable in the wild. The environment is also a laboratory for controlled experiments by which we can analyze attribution methods and suggest improvements.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
On Understanding the Influence of Controllable Factors with a Feature Attribution Algorithm: a Medical Case Study
Feature attribution XAI algorithms enable their users to gain insight into the underlying patterns of large datasets through their feature importance calculation. Existing feature attribution algorithms treat all feature…
Explainable Artificial Intelligence (XAI)Feature ImportanceTowards a Theory of Faithfulness: Faithful Explanations of Differentiable Classifiers over Continuous Data
There is broad agreement in the literature that explanation methods should be faithful to the model that they explain, but faithfulness remains a rather vague term. We revisit faithfulness in the context of continuous da…
PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution
Visual attribution is a fundamental tool for interpreting modern vision and vision-language models, particularly when their decisions must be inspected, diagnosed, or audited. Its goal is to explain how a model's decisio…
Image ClassificationImage CaptioningObject DetectionVisual GroundingA Dual-Perspective Approach to Evaluating Feature Attribution Methods
Feature attribution methods attempt to explain neural network predictions by identifying relevant features. However, establishing a cohesive framework for assessing feature attribution remains a challenge. There are seve…
Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainability
Deep neural network predictions are notoriously difficult to interpret. Feature attribution methods aim to explain these predictions by identifying the contribution of each input feature. Faithfulness, often evaluated us…