The Expected Jacobian Outerproduct: Theory and Empirics
The expected gradient outerproduct (EGOP) of an unknown regression function is an operator that arises in the theory of multi-index regression, and is known to recover those directions that are most relevant to predicting the output. However, work on the EGOP, including that on its cheap estimators, is restricted to the regression setting. In this work, we adapt this operator to the multi-class setting, which we dub the expected Jacobian outerproduct (EJOP). Moreover, we propose a simple rough estimator of the EJOP and show that somewhat surprisingly, it remains statistically consistent under mild assumptions. Furthermore, we show that the eigenvalues and eigenspaces also remain consistent. Finally, we show that the estimated EJOP can be used as a metric to yield improvements in real-world non-parametric classification tasks: both by its use as a metric, and also as cheap initialization in metric learning tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Metric LearningregressionSimilar Papers 제목 키워드 기반
An Equilibrium Model of the First-Price Auction with Strategic Uncertainty: Theory and Empirics
In many first-price auctions, bidders face considerable strategic uncertainty: They cannot perfectly anticipate the other bidders' bidding behavior. We propose a model in which bidders do not know the entire distribution…
Optimal Post-Hoc Theorizing
For many economic questions, the empirical results are not interesting unless they are strong. For these questions, theorizing before the results are known is not always optimal. Instead, the optimal sequencing of theory…
Supervised Learning Has a Geometric Blind Spot
Ordinary supervised training minimises the task loss and then stops. It never pays for how far the representation moves when the input is nudged along directions that helped fit training labels---including directions tha…
Deep Learning Based Residuals in Non-linear Factor Models: Precision Matrix Estimation of Returns with Low Signal-to-Noise Ratio
This paper introduces a consistent estimator and rate of convergence for the precision matrix of asset returns in large portfolios using a non-linear factor model within the deep learning framework. Our estimator remains…
validAlmost Sure Asymptotic Freeness of Neural Network Jacobian with Orthogonal Weights
A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobia…