BEGIN:VCALENDAR VERSION:2.0 PRODID:researchseminars.org CALSCALE:GREGORIAN X-WR-CALNAME:researchseminars.org BEGIN:VEVENT SUMMARY:Chris Budd OBE (University of Bath) DTSTART:20260710T223000Z DTEND:20260710T233000Z DTSTAMP:20260625T105605Z UID:AppliedMath/94 DESCRIPTION:Title: Equidistribution-based training of univariate free knot splines\, PIN NS\, and ReLU neural networks\nby Chris Budd OBE (University of Bath) as part of SFU Mathematics of Computation\, Application and Data ("MOCAD") Seminar\n\nInteractive livestream: https://sfu.zoom.us/j/88232824688?pwd= SSwf2Nk28PAmzRguQcYrdLYaXKHml9.1\nLecture held in K9509.\n\nAbstract\nWe c onsider the problem of improving the accuracy\, convergence\, and conditio ning of univariate nonlinear function approximations using (mainly) shallo w neural networks (NN) with a rectified linear unit (ReLU) activation func tion. The standard L_2 based approximation problem is ill-conditioned and the behaviour of the optimisation algorithms used in training these networ ks degrades rapidly as the width of the network increases. This can lead t o significantly poorer approximation in practice than we would expect from the theoretical expressivity of the ReLU NN architecture. Univariate shal low ReLU NNs and traditional approximation methods\, such as univariate F ree Knot Splines (FKS) span the same function space\, and thus have the sa me theoretical expressivity. \n\nHowever\, the FKS representation\, both r emains well-conditioned as the number of knots increases\, and can be high ly accurate if the knots are correctly placed. We leverage the theory of o ptimal piecewise linear interpolants to improve the training procedure for both a FKS and a ReLU NN. For the FKS we propose a novel two-level trai ning procedure. First solving the nonlinear problem of finding the optimal knot locations of the interpolating FKS using an equidistribution approac h. Then solving the nearly linear\, well-conditioned\, problem of finding the optimal weights and knots of the FKS. \n\nThe training of the FKS give s insights into how we can train a ReLU NN effectively to give an equally accurate approximation. To do this we combine the training of the ReLU NN with an equidistribution based loss to find the breakpoints of the ReLU fu nctions\, this is then combined with preconditioning the ReLU NN approxima tion (to take an FKS form) to find the scalings of the ReLU\, functions. T his procedure leads to a fast\, well-conditioned and reliable method of fi nding an accurate shallow ReLU NN approximation to a univariate target fun ction. This method avoids spectral bias and is highly effective for a wide variety of functions. We test this method on a series of regular\, singul ar\, and rapidly varying target functions and obtain good results\, realis ing the expressivity of the shallow ReLU network in all cases. We conclude that in the shallow case to gain full expressivity for the ReLU NN we mu st both find the optimal breakpoints (by equidistribution) and preconditio n the problem of finding the optimal coefficients. We then extend our resu lts to more general activation functions\, and to deeper networks.\n\nWe t hen apply this methodology to the PINNS and DRM Machine learning methods f or solving differential equations\, showing that this leads to more accura te and stable schemes.\n LOCATION:https://researchseminars.org/talk/AppliedMath/94/ URL:https://sfu.zoom.us/j/88232824688?pwd=SSwf2Nk28PAmzRguQcYrdLYaXKHml9.1 END:VEVENT END:VCALENDAR