Negative eigenvalues of the Hessian in deep neural networks
The loss function of deep networks is known to be non-convex but the precise nature of this nonconvexity is still an active area of research. In this work, we study the loss landscape of deep networks through the eigendecompositions of their Hessian matrix. In particular, we examine how important the negative eigenvalues are and the benefits one can observe in handling them appropriately.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Understanding Memorization in Generative Models via Sharpness in Probability Landscapes
In this paper, we introduce a geometric framework to analyze memorization in diffusion models using the eigenvalues of the Hessian of the log probability density. We propose that memorization arises from isolated points …
MemorizationWinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is its slow convergence and early performan…
Community detection with the Bethe-Hessian
The Bethe-Hessian matrix, introduced by Saade, Krzakala, and Zdeborov\'a (2014), is a Hermitian matrix designed for applying spectral clustering algorithms to sparse networks. Rather than employing a non-symmetric and hi…
Community DetectionStochastic Block ModelTowards understanding the true loss surface of deep neural networks using random matrix theory and iterative spectral methods
The geometric properties of loss surfaces, such as the local flatness of a solution, are associated with generalization in deep learning. The Hessian is often used to understand these geometric properties. We investigate…
GPUAn analysis on negative curvature induced by singularity in multi-layer neural-network learning
In the neural-network parameter space, an attractive field is likely to be induced by singularities. In such a singularity region, first-order gradient learning typically causes a long plateau with very little change i…