The Hessian of tall-skinny networks is easy to invert
We describe an exact algorithm to solve linear systems of the form $Hx=b$ where $H$ is the Hessian of a deep net. The method computes Hessian-inverse-vector products without storing the Hessian or its inverse. It requires time and storage that scale linearly in the number of layers. This is in contrast to the naive approach of first computing the Hessian, then solving the linear system, which takes storage and time that are respectively quadratic and cubic in the number of layers. The Hessian-inverse-vector product method scales roughly like Pearlmutter's algorithm for computing Hessian-vector products.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Scalable methods for nonnegative matrix factorizations of near-separable tall-and-skinny matrices
Numerous algorithms are used for nonnegative matrix factorization under the assumption that the matrix is nearly separable. In this paper, we show how to make these algorithms efficient for data matrices that have many m…
Projected Tensor-Tensor Products for Efficient Computation of Optimal Multiway Data Representations
Tensor decompositions have become essential tools for feature extraction and compression of multiway data. Recent advances in tensor operators have enabled desirable properties of standard matrix algebra to be retained f…
PyTorch CurveBall - A second-order optimizer for deep networks
We propose a fast second-order method that can be used as a drop-in replacementfor current deep learning solvers. Compared to stochastic gradient descent (SGD),it only requires two additional forward-mode automatic diffe…
Small steps and giant leaps: Minimal Newton solvers for Deep Learning
We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic dif…
End-to-end Feature Selection Approach for Learning Skinny Trees
We propose a new optimization-based approach for feature selection in tree ensembles, an important problem in statistics and machine learning. Popular tree ensemble toolkits e.g., Gradient Boosted Trees and Random Forest…
Ensemble LearningFeature CompressionFeature Importancefeature selection+1