Be Careful When Evaluating Explanations Regarding Ground Truth
Evaluating explanations of image classifiers regarding ground truth, e.g. segmentation masks defined by human perception, primarily evaluates the quality of the models under consideration rather than the explanation methods themselves. Driven by this observation, we propose a framework for $\textit{jointly}$ evaluating the robustness of safety-critical systems that $\textit{combine}$ a deep neural network with an explanation method. These are increasingly used in real-world applications like medical image analysis or robotics. We introduce a fine-tuning procedure to (mis)align model$\unicode{x2013}$explanation pipelines with ground truth and use it to quantify the potential discrepancy between worst and best-case scenarios of human alignment. Experiments across various model architectures and post-hoc local interpretation methods provide insights into the robustness of vision transformers and the overall vulnerability of such AI systems to potential adversarial attacks.
Code (1)
Tasks
Medical Image AnalysisSimilar Papers 제목 키워드 기반
Evaluating the Utility of Model Explanations for Model Development
One of the motivations for explainable AI is to allow humans to make better and more informed decisions regarding the use and deployment of AI models. But careful evaluations are needed to assess whether this expectation…
counterfactualDecision MakingmodelModel SelectionGuiding the generation of counterfactual explanations through temporal background knowledge for Predictive Process Monitoring
Counterfactual explanations suggest what should be different in the input instance to change the outcome of an AI system. When dealing with counterfactual explanations in the field of Predictive Process Monitoring, howev…
counterfactualPredictive Process Monitoring(When) Are Contrastive Explanations of Reinforcement Learning Helpful?
Global explanations of a reinforcement learning (RL) agent's expected behavior can make it safer to deploy. However, such explanations are often difficult to understand because of the complicated nature of many RL polici…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Evaluating Model Explanations without Ground Truth
There can be many competing and contradictory explanations for a single model prediction, making it difficult to select which one to use. Current explanation evaluation frameworks measure quality by comparing against ide…
Feature ImportancemodelSensitivityEvaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set
Explainable artificial intelligence (XAI) is concerned with producing explanations indicating the inner workings of models. For a Rashomon set of similarly performing models, explanations provide a way of disambiguating …