A Variational Model for Automatic Differentiation with Applications to Deep Learning

Abstract: Automatic differentiation is an automatized implementation of differential calculus, it plays a key computational role in several fields as machine learning, design optimization, fluid dynamics, physical modeling, mechanics, finance. It is also efficient for nonsmooth problems despite the occurence of spurious behaviors. In that case, one indeed observes the apparition of calculus artifacts and artificial critical points that have no variational nature. Our goal is to provide a simple mathematical model for this differentiation process. Our motivation comes from deep learning which will also serve as an illustrative model for our ideas and results. The first easy, but somehow unexpected fact, is that there is no «subdifferentiation» operator modeling nonsmooth nonconvex automatic differentiation. This fact motivates the introduction of a family of multivalued mappings generalizing gradient-like behaviors that we call conservative fields. We shall review their salient properties and show how they allow us to study rigorously forward and backward automatic differentiation. We will also try to clarify the spurious behavior of automatic differentiation and study the role of what we call «artificial critical points». We apply our findings to show that the training of feedforward neural networks through mini-batch stochastic «subgradient» methods comes with rigorous convergence guarantees. Joint work with E. Pauwels

optimization and control

Audience: researchers in the topic

Comments: the address and password of the zoom room of the seminar are sent by e-mail on the mailinglist of the seminar one day before each talk

---

One World Optimization seminar

Series comments: Description: Online seminar on optimization and related areas

The address and password of the zoom room of the seminar are sent by e-mail on the mailinglist of the seminar one day before each talk

---