A note on regularised NTK dynamics with an application to PAC-Bayesian training

Clerico, Eugenio; Guedj, Benjamin

Document type :

Pré-publication ou Document de travail

Title :

A note on regularised NTK dynamics with an application to PAC-Bayesian training

Author(s) :

Clerico, Eugenio [Auteur]
Guedj, Benjamin [Auteur]

MOdel for Data Analysis and Learning [MODAL]
The Inria London Programme [Inria-London]
The Alan Turing Institute
Inria Lille - Nord Europe
Institut National de Recherche en Informatique et en Automatique [Inria]
Department of Computer science [University College of London] [UCL-CS]
University College of London [London] [UCL]

Publication date :

2023-12-20

HAL domain(s) :

Statistiques [stat]/Machine Learning [stat.ML]
Informatique [cs]/Apprentissage [cs.LG]

English abstract : [en]

We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, ...
Show more >We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, where the dynamics can be linearised around the initialisation. The standard neural tangent kernel (NTK) governs the evolution during the training in the infinite-width limit, although the regularisation yields an additional term appears in the differential equation describing the dynamics. This setting provides an appropriate framework to study the evolution of wide networks trained to optimise generalisation objectives such as PAC-Bayes bounds, and hence potentially contribute to a deeper theoretical understanding of such networks.Show less >

Language :

Anglais

Collections :

Laboratoire Paul Painlevé - UMR 8524

Source :

Harvested from HAL

Files

document
Open access
Access the document

2312.13259.pdf
Open access
Access the document

2312.13259
Open access
Access the document

A note on regularised NTK dynamics with ... BibTeX CSV Excel RIS

Files

A note on regularised NTK dynamics with ...

BibTeX

CSV

Excel

RIS