Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case
The paper deals with the distribution of singular values of the input-output Jacobian of deep untrained neural networks in the limit of their infinite width. The Jacobian is the product of random matrices where the independent rectangular weight matrices alternate with diagonal matrices whose entries depend on the corresponding column of the nearest neighbor weight matrix. The problem was considered in \cite{Pe-Co:18} for the Gaussian weights and biases and also for the weights that are Haar distributed orthogonal matrices and Gaussian biases. Basing on a free probability argument, it was claimed that in these cases the singular value distribution of the Jacobian in the limit of infinite width (matrix size) coincides with that of the analog of the Jacobian with special random but weight independent diagonal matrices, the case well known in random matrix theory. The claim was rigorously proved in \cite{Pa-Sl:21} for a quite general class of weights and biases with i.i.d. (including Gaussian) entries by using a version of the techniques of random matrix theory. In this paper we use another version of the techniques to justify the claim for random Haar distributed weight matrices and Gaussian biases.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Statistics of Min-max Normalized Eigenvalues in Random Matrices
Random matrix theory has played an important role in various areas of pure mathematics, mathematical physics, and machine learning. From a practical perspective of data science, input data are usually normalized prior to…
Estimating the Spectral Density of Large Implicit Matrices
Many important problems are characterized by the eigenvalues of a large matrix. For example, the difficulty of many optimization problems, such as those arising from the fitting of large models in statistics and machine …
Eigenvalues of Autoencoders in Training and at Initialization
In this paper, we investigate the evolution of autoencoders near their initialization. In particular, we study the distribution of the eigenvalues of the Jacobian matrices of autoencoders early in the training process, t…
Eigenvalue distribution of nonlinear models of random matrices
This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider $M= \frac{1}{m} YY^*$ with $Y=f(WX)$ where $W$ and $X$ are random rectangul…
Optimal Filter Design for Consensus on Random Directed Graphs
Optimal design of consensus acceleration graph filters relates closely to the eigenvalues of the consensus iteration matrix. This task is complicated by random networks with uncertain iteration matrix eigenvalues. Filter…