Rethinking Attention-Model Explainability through Faithfulness Violation Test
Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading -- features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as Attentio$\odot$Gradient and LRP-based attention explanations. We first propose an actionable diagnostic methodology (henceforth faithfulness violation test) to measure the consistency between explanation weights and the impact polarity. Through the extensive experiments, we then show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue, especially the raw attention. Empirical analyses on the factors affecting violation issues further provide useful observations for adopting explanation methods in attention models.
Code (1)
Tasks
DiagnosticFeature ImportancemodelSimilar Papers 제목 키워드 기반
Explainability in Generative Medical Diffusion Models: A Faithfulness-Based Analysis on MRI Synthesis
This study investigates the explainability of generative diffusion models in the context of medical imaging, focusing on Magnetic resonance imaging (MRI) synthesis. Although diffusion models have shown strong performance…
Decision MakingThe Explanation Game: Towards Prediction Explainability through Sparse Communication
Explainability is a topic of growing importance in NLP. In this work, we provide a unified perspective of explainability as a communication problem between an explainer and a layperson about a classifier's decision. We u…
feature selectionMachine TranslationPredictiontext-classification+2Disagreeing Rationales: Rethinking Classification and Explainability Evaluation in Hate Speech Detection
Human disagreement is ubiquitous and well-known in labeling. However, variation in explanations, captured through token-level human rationales, remains far less explored. At the same time, it is unclear how to best evalu…
Hate Speech DetectionListening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models
Transformer-based automatic speech recognition (ASR) models such as Whisper are highly accurate, but their predictions remain difficult to interpret. Existing explainable AI (XAI) methods often lack faithfulness and prec…
Speech RecognitionCAMAL: Improving Attention Alignment and Faithfulness with Segmentation Masks
Many vision datasets now provide segmentation masks in addition to annotated images to support a wide range of tasks. In this work, we propose Class Activation Map Attention Learning (CAMAL), an efficient and scalable me…
Reinforcement Learning