Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one of them, the so-called Covariance of gradients. We show this term is not a Covariance but a second moment matrix, i.e., it is influenced by means of gradients. These means possess an additive two-way structure that is the source of the outliers in the spectrum. This structure can be used to approximate the principal subspace of the Hessian using certain "averaging" operations, avoiding the need for high-dimensional eigenanalysis. We corroborate this claim across different datasets, architectures and sample sizes.
Code (1)
Similar Papers 제목 키워드 기반
Hierarchical Reference Sets for Robust Unsupervised Detection of Scattered and Clustered Outliers
Most real-world IoT data analysis tasks, such as clustering and anomaly event detection, are unsupervised and highly susceptible to the presence of outliers. In addition to sporadic scattered outliers caused by factors s…
Outlier DetectionStructured Sampling for Robust Euclidean Distance Geometry
This paper addresses the problem of estimating the positions of points from distance measurements corrupted by sparse outliers. Specifically, we consider a setting with two types of nodes: anchor nodes, for which exact d…
Robust Bayesian Compressed sensing
We consider the problem of robust compressed sensing whose objective is to recover a high-dimensional sparse signal from compressed measurements corrupted by outliers. A new sparse Bayesian learning method is developed f…
compressed sensingTrust No One: Low Rank Matrix Factorization Using Hierarchical RANSAC
In this paper we present a system for performing low rank matrix factorization. Low-rank matrix factorization is an essential problem in many areas including computer vision, with applications in e.g. affine structure-fr…
Hierarchical topological clustering
Topological methods have the potential of exploring data clouds without making assumptions on their the structure. Here we propose a hierarchical topological clustering algorithm that can be implemented with any distance…