Position: Curvature Matrices Should Be Democratized via Linear Operators
Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed $\textit{curvlinops}$, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with $\textit{curvlinops}$ how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs.
Code (0)
등록된 구현이 없습니다.
Tasks
PositionUncertainty QuantificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks
Second-order optimization approaches like the generalized Gauss-Newton method are considered more powerful as they utilize the curvature information of the objective function with preconditioning matrices. Albeit offerin…
Near-Field Joint Localization and Synchronization
In 5G communication, linear and planar arrays are used for both positioning and communication. As the arrays grow in size, the conventional far-field assumption is increasingly being violated and curvature of the wavefro…
The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network
An important factor contributing to the success of deep learning has been the remarkable ability to optimize large neural networks using simple first-order optimization algorithms like stochastic gradient descent. While …
A Mean Curvature Approach to Boundary Detection: Geometric Insights for Unsupervised Learning
Accurate boundary detection in high-dimensional data remains a central challenge in unsupervised learning, particularly in the presence of non-linear structures and heterogeneous densities. In this work, we introduce Mea…
Boundary DetectionModular Block-diagonal Curvature Approximations for Feedforward Architectures
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and po…
BIG-bench Machine Learning