The asymptotic spectrum of the Hessian of DNN throughout training

Tuesday, 1 October, 2019

Published in: 

arXiv:1910.02875

The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs: we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training.

Author(s): 

Arthur Jacot
Franck Gabriel
Clément Hongler