Universal characteristics of deep neural network loss surfaces from random matrix theory
This paper considers several aspects of random matrix universality in deep neural networks. Motivated by recent experimental work, we use universal properties of random matrices related to local statistics to derive practical implications for deep neural networks based on a realistic model of their Hessians. In particular we derive universal aspects of outliers in the spectra of deep neural networks and demonstrate the important role of random matrix local laws in popular pre-conditioning gradient descent algorithms. We also present insights into deep neural network loss surfaces from quite general arguments based on tools from statistical physics and random matrix theory.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Random matrix theory and the loss surfaces of neural networks
Neural network models are one of the most successful approaches to machine learning, enjoying an enormous amount of development and research over recent years and finding concrete real-world applications in almost any co…
Expressive Power and Loss Surfaces of Deep Learning Models
The goals of this paper are two-fold. The first goal is to serve as an expository tutorial on the working of deep learning models which emphasizes geometrical intuition about the reasons for success of deep learning. The…
Deep LearningAppearance of Random Matrix Theory in Deep Learning
We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover excellent agreement with Gaussian Orthogonal Ensemble statistics across several network architect…
Deep LearningA spin-glass model for the loss surfaces of generative adversarial networks
We present a novel mathematical model that seeks to capture the key design feature of generative adversarial networks (GANs). Our model consists of two interacting spin glasses, and we conduct an extensive theoretical an…
Geometry of Neural Network Loss Surfaces via Random Matrix Theory
Understanding the geometry of neural network loss surfaces is important for the development of improved optimization algorithms and for building a theoretical understanding of why deep learning works. In this paper,…
Memorization