On Random Matrices Arising in Deep Neural Networks

Abstract: We study the distribution of singular values of product of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices. However, an important dierence is that the analog the of the population covariance matrices, assumed to be non-random or random but independent of the random data matrix in statistics and random matrix theory, are now certain functions of random data matrices (synaptic weight matrices in the deep neural network terminology). For the Gaussian synaptic weight matrices the problem has been treated in recent work [1] and certain subsequent works by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability to this case has to be justi ed. We use a version of the techniques of random matrix theory to justify and generalize the results of [1] to the case where the entries of the synaptic weight matrices are just independent identically distributed random variables with zero mean and nite fourth moment [2]. This, in particular, extends the property of the so-called macroscopic universality to the considered random matrices.

[1] J. Pennington, S. Schoenholz, and S. Ganguli, The emergence of spectral universality In: Proc. Mach. Learn. Res. (PMLR 70) 84 (2018) 1924-1932, arxiv.org/abs/1802.09979

[2] L. Pastur and V. Slavin, On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case, arxiv.org/abs/2011.11439.

statistical mechanicsmathematical physicsprobability

Audience: researchers in the topic

Séminaire MEGA

Series comments: Description: Monthly seminar on random matrices and random graphs