ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks. Existing methods based on implicit multiplication via automatic differentiation or Kronecker-factored block diagonal approximations do not consider noise in the mini-batch. We present ViViT, a curvature model that leverages the GGN's low-rank structure without further approximations. It allows for efficient computation of eigenvalues, eigenvectors, as well as per-sample first- and second-order directional derivatives. The representation is computed in parallel with gradients in one backward pass and offers a fine-grained cost-accuracy trade-off, which allows it to scale. We demonstrate this by conducting performance benchmarks and substantiate ViViT's usefulness by studying the impact of noise on the GGN's structural properties during neural network training.
Code (5)
Similar Papers 제목 키워드 기반
Fast Gauss-Newton for Multiclass Cross-Entropy
In multiclass softmax cross-entropy, the full generalized Gauss-Newton (GGN) curvature couples all output logits through the softmax covariance, making curvature-vector products harder to scale as the number of classes g…
Binary ClassificationOn the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width
The Hessian of neural networks can be decomposed into a sum of two matrices: (i) the positive semidefinite generalized Gauss-Newton matrix G, and (ii) the matrix H containing negative eigenvalues. We observe that for wid…
RelationGeodesic Exponential Kernels: When Curvature and Linearity Conflict
We consider kernel methods on general geodesic metric spaces and provide both negative and positive results. First we show that the common Gaussian kernel can only be generalized to a positive definite kernel on a geodes…
The impact of surface area, volume, curvature and Lennard-Jones potential to solvation modeling
This paper explores the impact of surface area, volume, curvature and Lennard-Jones potential on solvation free energy predictions. Rigidity surfaces are utilized to generate robust analytical expressions for maximum, mi…
Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks
Second-order optimization approaches like the generalized Gauss-Newton method are considered more powerful as they utilize the curvature information of the objective function with preconditioning matrices. Albeit offerin…