Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications
We develop techniques to quantify the degree to which a given (training or testing) example is an outlier in the underlying distribution. We evaluate five methods to score examples in a dataset by how well-represented the examples are, for different plausible definitions of "well-represented", and apply these to four common datasets: MNIST, Fashion-MNIST, CIFAR-10, and ImageNet. Despite being independent approaches, we find all five are highly correlated, suggesting that the notion of being well-represented can be quantified. Among other uses, we find these methods can be combined to identify (a) prototypical examples (that match human expectations); (b) memorized training examples; and, (c) uncommon submodes of the dataset. Further, we show how we can utilize our metrics to determine an improved ordering for curriculum learning, and impact adversarial robustness. We release all metric values on training and test sets we studied.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial RobustnessBIG-bench Machine LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Normal Variance Mixture Model for Robust Kalman Filtering
The Kalman filter is ubiquitous for state space models because of its desirable statistical properties, ease of implementation, and generally good performance. However, it can perform poorly in the presence of outliers, …
State Space ModelsDensity-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion
Chamfer Distance (CD) and Earth Mover's Distance (EMD) are two broadly adopted metrics for measuring the similarity between two point sets. However, CD is usually insensitive to mismatched local density, and EMD is usual…
Point Cloud CompletionBalanced Chamfer Distance as a Comprehensive Metric for Point Cloud Completion
Chamfer Distance (CD) and Earth Mover’s Distance (EMD) are two broadly adopted metrics for measuring the similarity between two point sets. However, CD is usually insensitive to mismatched local density, and EMD is usual…
Point Cloud CompletionAsymmetry in Distributions of Accumulated Gains and Losses in Stock Returns
We study decades-long historic distributions of accumulated S\&P500 returns, from daily returns to those over several weeks. The time series of the returns emphasize major upheavals in the markets -- Black Monday, Tech B…
Reliable Fidelity and Diversity Metrics for Generative Models
Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fr\'echet Inception Di…
DiversityImage Generation