BEGIN:VCALENDAR VERSION:2.0 PRODID:researchseminars.org CALSCALE:GREGORIAN X-WR-CALNAME:researchseminars.org BEGIN:VEVENT SUMMARY:Yasaman Bahri (Google Brain) DTSTART:20200701T171500Z DTEND:20200701T174000Z DTSTAMP:20260423T022812Z UID:SciDL/10 DESCRIPTION:Title: L earning Dynamics of Wide\, Deep Neural Networks: Beyond the Limit of Infin ite Width\nby Yasaman Bahri (Google Brain) as part of Workshop on Scie ntific-Driven Deep Learning (SciDL)\n\n\nAbstract\nWhile many practical ad vancements in deep learning have been made in recent years\, a scientific\ , and ideally theoretical\, understanding of modern neural networks is sti ll in its infancy. At the heart of this would be to better understand the learning dynamics of such systems. In a first step towards tackling this p roblem\, one can try to identify limits that have theoretical tractability and are potentially practically relevant. I’ll begin by surveying our b ody of work that has investigated the infinite width limit of deep network s. These results establish exact mappings between deep networks and other\ , existing machine learning methods (namely\, Gaussian processes and kerne l methods) but with novel modifications to them that had not been previous ly encountered. With these exact mappings in hand\, the natural question i s to what extent they bear relevance to neural networks at finite width. I ’ll argue that the choice of learning rate is a crucial factor in dynami cs away from this limit and naturally classifies deep networks into two cl asses separated by a sharp phase transition. This is elucidated in a class of solvable simple models we present\, which give quantitative prediction s for the two phases. Quite remarkably\, we test these empirically in prac tical settings and find excellent agreement.\n LOCATION:https://researchseminars.org/talk/SciDL/10/ END:VEVENT END:VCALENDAR