TimeSHAP: Explaining Recurrent Models through Sequence Perturbations
Recurrent neural networks are a standard building block in numerous machine learning domains, from natural language processing to time-series classification. While their application has grown ubiquitous, understanding of their inner workings is still lacking. In practice, the complex decision-making in these models is seen as a black-box, creating a tension between accuracy and interpretability. Moreover, the ability to understand the reasoning process of a model is important in order to debug it and, even more so, to build trust in its decisions. Although considerable research effort has been guided towards explaining black-box models in recent years, recurrent models have received relatively little attention. Any method that aims to explain decisions from a sequence of instances should assess, not only feature importance, but also event importance, an ability that is missing from state-of-the-art explainers. In this work, we contribute to filling these gaps by presenting TimeSHAP, a model-agnostic recurrent explainer that leverages KernelSHAP's sound theoretical footing and strong empirical results. As the input sequence may be arbitrarily long, we further propose a pruning method that is shown to dramatically improve its efficiency in practice.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingFeature ImportanceTime SeriesTime Series AnalysisTime Series ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TimeSHAP: Explaining Recurrent Models through Sequence Perturbations
Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agno…
Decision MakingFeature ImportanceFraud DetectionSequential Decision Making+2WindowSHAP: An Efficient Framework for Explaining Time-series Classifiers based on Shapley Values
Unpacking and comprehending how black-box machine learning algorithms make decisions has been a persistent challenge for researchers and end-users. Explaining time-series predictive models is useful for clinical applicat…
CPUTime SeriesTime Series AnalysisA causal framework for explaining the predictions of black-box sequence-to-sequence models
We interpret the predictions of any black-box structured input-structured output model around a specific input-output pair. Our method returns an "explanation" consisting of groups of input-output tokens that are causall…
Dynamically Computing Adversarial Perturbations for Recurrent Neural Networks
Convolutional and recurrent neural networks have been widely employed to achieve state-of-the-art performance on classification tasks. However, it has also been noted that these networks can be manipulated adversarially …
Learning Decision Trees Recurrently Through Communication
Integrated interpretability without sacrificing the prediction accuracy of decision making algorithms has the potential of greatly improving their value to the user. Instead of assigning a label to an image directly, we …
Decision Makingimage-classificationImage Classification