Free Probability for predicting the performance neural networks

Abstract: Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing stability. Following the works of Pennington et al., such Jacobians are modeled using free multiplicative convolutions from Free Probability Theory (FPT). We make the following contributions: – theoretical: refine the metamodel of Pennington et al. thanks to the rectangular analogue of free multiplicative convolutions. – numerical: present and benchmark a homotopy method for solving the equations of free probability. – empirical: we show that the relevant FPT metrics computed before training are highly correlated to final test accuracies – up to 85%.

mathematical physicscombinatoricsprobabilitystatistics theory

Audience: researchers in the topic

Abu Dhabi Stochastics Seminar