BEGIN:VCALENDAR VERSION:2.0 PRODID:researchseminars.org CALSCALE:GREGORIAN X-WR-CALNAME:researchseminars.org BEGIN:VEVENT SUMMARY:Jakub Konečný (Google) DTSTART:20200708T130000Z DTEND:20200708T140000Z DTSTAMP:20260423T035931Z UID:FLOW/8 DESCRIPTION:Title: On the outsized importance of learning rates in local update methods\nby Jakub Konečný (Google) as part of Federated Learning One World Seminar\n \n\nAbstract\nIn this work\, we study a family of algorithms\, which we re fer to as local update methods\, that generalize many federated learning a nd meta-learning algorithms. We prove that for quadratic objectives\, loca l update methods perform stochastic gradient descent on a surrogate loss f unction which we exactly characterize. We show that the choice of client l earning rate controls the condition number of that surrogate loss\, as wel l as the distance between the minimizers of the surrogate and true loss fu nctions. We use this theory to derive novel convergence rates for federate d averaging that showcase this trade-off between the condition number of t he surrogate loss and its alignment with the true loss function. We valida te our results empirically\, showing that in communication-limited setting s\, proper learning rate tuning is often sufficient to reach near-optimal behavior. We also present a practical method for automatic learning rate d ecay in local update methods that helps reduce the need for learning rate tuning\, and highlight its empirical performance on a variety of tasks and datasets.\n LOCATION:https://researchseminars.org/talk/FLOW/8/ END:VEVENT END:VCALENDAR