BEGIN:VCALENDAR VERSION:2.0 PRODID:researchseminars.org CALSCALE:GREGORIAN X-WR-CALNAME:researchseminars.org BEGIN:VEVENT SUMMARY:Suriya Gunasekar (Microsoft Research) DTSTART:20210409T150000Z DTEND:20210409T161200Z DTSTAMP:20260423T022712Z UID:sss/23 DESCRIPTION:Title: Fun ctions space view of linear multi-channel convolution networks with bounde d weight norm\nby Suriya Gunasekar (Microsoft Research) as part of Sto chastics and Statistics Seminar Series\n\n\nAbstract\nThe magnitude of the weights of a neural network is a fundamental measure of complexity that p lays a crucial role in the study of implicit and explicit regularization. For example\, in recent work\, gradient descent updates in overparameteriz ed models asymptotically lead to solutions that implicitly minimize the el l_2 norm of the parameters of the model\, resulting in an inductive bias t hat is highly architecture dependent. To investigate the properties of lea rned functions\, it is natural to consider a function space view given by the minimum ell_2 norm of weights required to realize a given function wit h a given network. We call this the “induced regularizer” of the netwo rk. Building on a line of recent work\, we study the induced regularizer o f linear convolutional neural networks with a focus on the role of kernel size and the number of channels. We introduce an SDP relaxation of the ind uced regularizer\, that we show is tight for networks with a single input channel. Using this SDP formulation\, we show that the induced regularizer is independent of the number of the output channels for single-input chan nel networks\, and for multi-input channel networks\, we show independence given sufficiently many output channels. Moreover\, we show that as the k ernel size increases\, the induced regularizer interpolates between a basi s-invariant norm and a basis-dependent norm that promotes sparse structure s in Fourier space.\n\nBased on joint work with Meena Jagadeesan and Ilya Razenshteyn.\n LOCATION:https://researchseminars.org/talk/sss/23/ END:VEVENT END:VCALENDAR