Interpreting Low-level Vision Models with Causal Effect Maps
Deep neural networks have significantly improved the performance of low-level vision tasks but also increased the difficulty of interpretability. A deep understanding of deep models is beneficial for both network design and practical reliability. To take up this challenge, we introduce causality theory to interpret low-level vision models and propose a model-/task-agnostic method called Causal Effect Map (CEM). With CEM, we can visualize and quantify the input-output relationships on either positive or negative effects. After analyzing various low-level vision tasks with CEM, we have reached several interesting insights, such as: (1) Using more information of input images (e.g., larger receptive field) does NOT always yield positive outcomes. (2) Attempting to incorporate mechanisms with a global receptive field (e.g., channel attention) into image denoising may prove futile. (3) Integrating multiple tasks to train a general model could encourage the network to prioritize local information over global context. Based on the causal effect theory, the proposed diagnostic tool can refresh our common knowledge and bring a deeper understanding of low-level vision models. Codes are available at https://github.com/J-FHu/CEM.
Code (1)
Tasks
DenoisingDiagnosticImage DenoisingSimilar Papers 제목 키워드 기반
Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs
Understanding how vision-language models (VLMs) interpret data visualizations remains an open problem, and is increasingly important as these models are used for analytical tasks where reliable reasoning is essential. We…
Answer GenerationText GenerationInterpreting TSLS Estimators in Information Provision Experiments
To estimate the causal effects of beliefs on actions, researchers often run information provision experiments. We consider the causal interpretation of two-stage least squares (TSLS) estimators in these experiments. We c…
Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps
Transformers are ubiquitous in wide tasks. Interpreting their internals is a pivotal goal. Nevertheless, their particular components, feed-forward (FF) blocks, have typically been less analyzed despite their substantial …
Learn to Rank: Visual Attribution by Learning Importance Ranking
Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attri…
OCCAM: Open-set Causal Concept explAnation and Ontology induction for black-box vision Models
Interpreting the decisions of deep image classifiers remains challenging, particularly in black-box settings where model internals are inaccessible. We introduce OCCAM, a framework for open-set causal concept explanation…
Image Attribution