Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity
The use of eXplainable Artificial Intelligence (XAI) systems has introduced a set of challenges that need resolution. The XAI robustness, or stability, has been one of the goals of the community from its beginning. Multiple authors have proposed evaluating this feature using objective evaluation measures. Nonetheless, many questions remain. With this work, we propose a novel approach to meta-evaluate these metrics, i.e. analyze the correctness of the evaluators. We propose two new tests that allowed us to evaluate two different stability measures: AVG-Sensitiviy and MAX-Senstivity. We tested their reliability in the presence of perfect and robust explanations, generated with a Decision Tree; as well as completely random explanations and prediction. The metrics results showed their incapacity of identify as erroneous the random explanations, highlighting their overall unreliability.
Code (1)
Tasks
AvgExplainable artificial intelligenceExplainable Artificial Intelligence (XAI)SensitivityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Beyond Aggregate Scores: Behavioral Correctness Assumptions for Assessing Reference-Based Automatic Evaluation Methods
Automated reference-based evaluation methods play a critical role in assessing natural language generation systems. Existing meta-evaluation primarily measures agreement with human judgments or benchmark labels, providin…
Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation
The correlation between NLG automatic evaluation metrics and human evaluation is often regarded as a critical criterion for assessing the capability of an evaluation metric. However, different grouping methods and correl…
nlg evaluationMANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continu…
Class Incremental LearningContinual LearningMeasuring the Stability of Process Outcome Predictions in Online Settings
Predictive Process Monitoring aims to forecast the future progress of process instances using historical event data. As predictive process monitoring is increasingly applied in online settings to enable timely interventi…
Decision MakingPredictive Process MonitoringTime SeriesA Sparse Polynomial Chaos Expansion-Based Method for Probabilistic Transient Stability Assessment and Enhancement
This paper proposes an adaptive sparse polynomial chaos expansion(PCE)-based method to quantify the impacts of uncertainties on critical clearing time (CCT) that is an important index in transient stability analysis. The…
Sensitivity