On Random Matrices Arising in Deep Neural Networks. Gaussian Case
The paper deals with distribution of singular values of product of random matrices arising in the analysis of deep neural networks. The matrices resemble the product analogs of the sample covariance matrices, however, an important difference is that the population covariance matrices, which are assumed to be non-random in the standard setting of statistics and random matrix theory, are now random, moreover, are certain functions of random data matrices. The problem has been considered in recent work [21] by using the techniques of free probability theory. Since, however, free probability theory deals with population matrices which are independent of the data matrices, its applicability in this case requires an additional justification. We present this justification by using a version of the standard techniques of random matrix theory under the assumption that the entries of data matrices are independent Gaussian random variables. In the subsequent paper [18] we extend our results to the case where the entries of data matrices are just independent identically distributed random variables with several finite moments. This, in particular, extends the property of the so-called macroscopic universality on the considered random matrices.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case
The paper deals with the distribution of singular values of the input-output Jacobian of deep untrained neural networks in the limit of their infinite width. The Jacobian is the product of random matrices where the indep…
On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case
We study the distribution of singular values of product of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important dif…
Additivity of Information in Multilayer Networks via Additive Gaussian Noise Transforms
Multilayer (or deep) networks are powerful probabilistic models based on multiple stages of a linear transform followed by a non-linear (possibly random) function. In general, the linear transforms are defined by matrice…
Quantitative deterministic equivalent of sample covariance matrices with a general dependence structure
We study sample covariance matrices arising from rectangular random matrices with i.i.d. columns. It was previously known that the resolvent of these matrices admits a deterministic equivalent when the spectral parameter…
Eigenvalue distribution of nonlinear models of random matrices
This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider $M= \frac{1}{m} YY^*$ with $Y=f(WX)$ where $W$ and $X$ are random rectangul…