Why You Should Not Trust Interpretations in Machine Learning: Adversarial Attacks on Partial Dependence Plots
The adoption of artificial intelligence (AI) across industries has led to the widespread use of complex black-box models and interpretation tools for decision making. This paper proposes an adversarial framework to uncover the vulnerability of permutation-based interpretation methods for machine learning tasks, with a particular focus on partial dependence (PD) plots. This adversarial framework modifies the original black box model to manipulate its predictions for instances in the extrapolation domain. As a result, it produces deceptive PD plots that can conceal discriminatory behaviors while preserving most of the original model's predictions. This framework can produce multiple fooled PD plots via a single model. By using real-world datasets including an auto insurance claims dataset and COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) dataset, our results show that it is possible to intentionally hide the discriminatory behavior of a predictor and make the black-box model appear neutral through interpretation tools like PD plots while retaining almost all the predictions of the original black-box model. Managerial insights for regulators and practitioners are provided based on the findings.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingManagementMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adversarial attacks and defenses in explainable artificial intelligence: A survey
Explainable artificial intelligence (XAI) methods are portrayed as a remedy for debugging and trusting statistical and deep learning models, as well as interpreting their predictions. However, recent advances in adversar…
Decision MakingExplainable artificial intelligenceExplainable Artificial Intelligence (XAI)Fairness+1Resilience of Bayesian Layer-Wise Explanations under Adversarial Attacks
We consider the problem of the stability of saliency-based explanations of Neural Network predictions under adversarial attacks in a classification task. Saliency interpretations of deterministic Neural Networks are rema…
General ClassificationAdversarial Training for Improving Model Robustness? Look at Both Prediction and Interpretation
Neural language models show vulnerability to adversarial examples which are semantically similar to their original counterparts with a few words replaced by their synonyms. A common way to improve model robustness is adv…
text-classificationText ClassificationAttacking interpretable NLP systems
Studies have shown that machine learning systems are vulnerable to adversarial examples in theory and practice. Where previous attacks have focused mainly on visual models that exploit the difference between human and ma…
Recent improvements of ASR models in the face of adversarial attacks
Like many other tasks involving neural networks, Speech Recognition models are vulnerable to adversarial attacks. However recent research has pointed out differences between attacks and defenses on ASR models compared to…
speech-recognitionSpeech Recognition