Investigating Saturation Effects in Integrated Gradients
Integrated Gradients has become a popular method for post-hoc model interpretability. De-spite its popularity, the composition and relative impact of different regions of the integral path are not well understood. We explore these effects and find that gradients in saturated regions of this path, where model output changes minimally, contribute disproportionately to the computed attribution. We propose a variant of IntegratedGradients which primarily captures gradients in unsaturated regions and evaluate this method on ImageNet classification networks. We find that this attribution technique shows higher model faithfulness and lower sensitivity to noise com-pared with standard Integrated Gradients. A note-book illustrating our computations and results is available at https://github.com/vivekmig/captum-1/tree/ExpandedIG.
Code (1)
Tasks
SensitivitySimilar Papers 제목 키워드 기반
Integrated Decision Gradients: Compute Your Attributions Where the Model Makes Its Decision
Attribution algorithms are frequently employed to explain the decisions of neural network models. Integrated Gradients (IG) is an influential attribution method due to its strong axiomatic foundation. The algorithm is ba…
FRInGe: Distribution-Space Integrated Gradients with Fisher--Rao Geometry
Gradient-based attribution methods are model-faithful and scalable, but Integrated Gradients (IG) can be brittle because explanations depend on heuristic baselines, straight-line paths, discretization, and saturation. We…
Expected Grad-CAM: Towards gradient faithfulness
Although input-gradients techniques have evolved to mitigate and tackle the challenges associated with gradients, modern gradient-weighted CAM approaches still rely on vanilla gradients, which are inherently susceptible …
counterfactualSensitivityAnglE-optimized Text Embeddings
High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding mo…
Language ModellingLarge Language ModelSemantic Textual SimilaritySentiment Analysis+1New possibilities in identification of binary choice models with fixed effects
We study the identification of binary choice models with fixed effects. We propose a condition called sign saturation and show that this condition is sufficient for identifying the model. In particular, this condition ca…