Explaining Anomalies using Denoising Autoencoders for Financial Tabular Data
Recent advances in Explainable AI (XAI) increased the demand for deployment of safe and interpretable AI models in various industry sectors. Despite the latest success of deep neural networks in a variety of domains, understanding the decision-making process of such complex models still remains a challenging task for domain experts. Especially in the financial domain, merely pointing to an anomaly composed of often hundreds of mixed type columns, has limited value for experts. Hence, in this paper, we propose a framework for explaining anomalies using denoising autoencoders designed for mixed type tabular data. We specifically focus our technique on anomalies that are erroneous observations. This is achieved by localizing individual sample columns (cells) with potential errors and assigning corresponding confidence scores. In addition, the model provides the expected cell value estimates to fix the errors. We evaluate our approach based on three standard public tabular datasets (Credit Default, Adult, IEEE Fraud) and one proprietary dataset (Holdings). We find that denoising autoencoders applied to this task already outperform other approaches in the cell error detection rates as well as in the expected value rates. Additionally, we analyze how a specialized loss designed for cell error detection can further improve these metrics. Our framework is designed for a domain expert to understand abnormal characteristics of an anomaly, as well as to improve in-house data quality management processes.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingDenoisingExplainable Artificial Intelligence (XAI)ManagementSimilar Papers 제목 키워드 기반
Fin-Fed-OD: Federated Outlier Detection on Financial Tabular Data
Anomaly detection in real-world scenarios poses challenges due to dynamic and often unknown anomaly distributions, requiring robust methods that operate under an open-world assumption. This challenge is exacerbated in pr…
Anomaly DetectionFederated LearningOutlier DetectionRepresentation LearningExplaining Anomalies Detected by Autoencoders Using SHAP
Anomaly detection algorithms are often thought to be limited because they don't facilitate the process of validating results performed by domain experts. In Contrast, deep learning algorithms for anomaly detection, such …
Anomaly DetectionOutlier DetectionDiffusion-Scheduled Denoising Autoencoders for Anomaly Detection in Tabular Data
Anomaly detection in tabular data remains challenging due to complex feature interactions and the scarcity of anomalous examples. Denoising autoencoders rely on fixed-magnitude noise, limiting adaptability to diverse dat…
Contrastive LearningAnomaly DetectionTracInAD: Measuring Influence for Anomaly Detection
As with many other tasks, neural networks prove very effective for anomaly detection purposes. However, very few deep-learning models are suited for detecting anomalies on tabular datasets. This paper proposes a novel me…
Anomaly DetectionMeasuring Privacy Risks and Tradeoffs in Financial Synthetic Data Generation
We explore the privacy-utility tradeoff of synthetic data generation schemes on tabular financial datasets, a domain characterized by high regulatory risk and severe class imbalance. We consider representative tabular da…
Synthetic Data Generation