How to Manipulate CNNs to Make Them Lie: the GradCAM Case
Recently many methods have been introduced to explain CNN decisions. However, it has been shown that some methods can be sensitive to manipulation of the input. We continue this line of work and investigate the explanation method GradCAM. Instead of manipulating the input, we consider an adversary that manipulates the model itself to attack the explanation. By changing weights and architecture, we demonstrate that it is possible to generate any desired explanation, while leaving the model's accuracy essentially unchanged. This illustrates that GradCAM cannot explain the decision of every CNN and provides a proof of concept showing that it is possible to obfuscate the inner workings of a CNN. Finally, we combine input and model manipulation. To this end we put a backdoor in the network: the explanation is correct unless there is a specific pattern present in the input, which triggers a malicious explanation. Our work raises new security concerns, especially in settings where explanations of models may be used to make decisions, such as in the medical domain.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Choose Your Explanation: A Comparison of SHAP and GradCAM in Human Activity Recognition
Explaining machine learning (ML) models using eXplainable AI (XAI) techniques has become essential to make them more transparent and trustworthy. This is especially important in high-stakes domains like healthcare, where…
Activity RecognitionFeature ImportanceHuman Activity RecognitionAttention-based Saliency Maps Improve Interpretability of Pneumothorax Classification
Purpose: To investigate chest radiograph (CXR) classification performance of vision transformers (ViT) and interpretability of attention-based saliency using the example of pneumothorax classification. Materials and Meth…
ClassificationLung Disease ClassificationEnhancing Explainable AI: A Hybrid Approach Combining GradCAM and LRP for CNN Interpretability
We present a new technique that explains the output of a CNN-based model using a combination of GradCAM and LRP methods. Both of these methods produce visual explanations by highlighting input regions that are important …
Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
The existence of redundancy in Convolutional Neural Networks (CNNs) enables us to remove some filters/channels with acceptable performance drops. However, the training objective of CNNs usually tends to minimize an accur…
Network PruningDiffGradCAM: A Universal Class Activation Map Resistant to Adversarial Training
Class Activation Mapping (CAM) and its gradient-based variants (e.g., GradCAM) have become standard tools for explaining Convolutional Neural Network (CNN) predictions. However, these approaches typically focus on indivi…