BEGIN:VCALENDAR VERSION:2.0 PRODID:researchseminars.org CALSCALE:GREGORIAN X-WR-CALNAME:researchseminars.org BEGIN:VEVENT SUMMARY:Jinshan Zeng/曾锦山 (Jiangxi Normal University) DTSTART:20201227T091500Z DTEND:20201227T100000Z DTSTAMP:20260423T041233Z UID:iccm2020/25 DESCRIPTION:Title: On ADMM in Deep Learning: Convergence and Saturation-Avoidance\nby J inshan Zeng/曾锦山 (Jiangxi Normal University) as part of ICCM 2020\n\n \nAbstract\nIn this talk\, we introduce an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type ac tivation functions (called sigmoid-ADMM pair)\, mainly motivated by the gr adient-free nature of ADMM in avoiding the saturation of sigmoid-type acti vations and the advantages of deep neural networks with sigmoid-type activ ations (called deep sigmoid nets) over their rectified linear unit (ReLU) counterparts (called deep ReLU nets) in terms of approximation. In particu lar\, we prove that the approximation capability of deep sigmoid nets is n ot worse than deep ReLU nets by showing that ReLU activation fucntion can be well approximated by deep sigmoid nets with two hidden layers and finit ely many free parameters but not vice-verse. We also establish the global convergence of the proposed ADMM for the nonlinearly constrained formulati on of the deep sigmoid nets training to a Karush-Kuhn-Tucker (KKT) point a t a rate of order O(1/k). Compared with the widely used stochastic gradien t descent (SGD) algorithm for the deep ReLU nets training (called ReLU-SGD pair)\, the proposed sigmoid-ADMM pair is practically stable with respect to the algorithmic hyperparameters including the learning rate\, initial schemes and the pro-processing of the input data. Moreover\, we find that to approximate and learn simple but important functions the proposed sigmo id-ADMM pair numerically outperforms the ReLU-SGD pair.\n LOCATION:https://researchseminars.org/talk/iccm2020/25/ END:VEVENT END:VCALENDAR