Theoretical Principles of Deep Learning

CentraleSupélec - M2 Maths-IA Université Paris-Saclay

Instructor: Hedi Hadiji

Reach me at: hedi.hadiji@l2s.centralesupelec.fr

Course Description

Machine learning algorithms involving deep neural networks have accumulated spectacular empirical successes over the recent years. Many of those accomplishments cannot be explained by conventional wisdom coming from standard learning theory. As the popularity of deep learning grows, the gap between theory and practice keeps widening. Building the groundworks for a satisfying theory of deep learning, with the ultimate goal of providing valuable insights to practitioners, is a major challenge in modern research.

In this class we will discuss recent theoretical progress made towards describing the empirical performance of deep methods. Our main focus will be the study of the surprisingly good generalization ability of deep networks.

Slides

Class 1: Introduction and Core Concepts. Approximation, Optimization and Generalization. Single-Layer Perceptron. [Slides]

Class 2: Approximation with Neural nets. Universality of approximation. Barron's theorem. [Slides]

Class 3: Lazy Training. Optimization in the lazy regime. Insights from overparameterized linear models. [Slides]

Class 4: Generalization. Concentration. Rademacher complexity. [Slides]

Class 5: The Neural Tangent Kernel. Definition and computation of Neural Tangent Kernels. Generalization bounds for the NTK. [Slides]

Class 6: PAC-Bayes Generalization bounds. PAC-Bayes bounds. MacAllester's bound. Non-vacuous bounds for stochastic networks. [Slides]

Class 7: Some mysteries. The Edge of stability. Feature learning.