Differentiable PAC-Bayes Objectives with ...
Type de document :
Compte-rendu et recension critique d'ouvrage
DOI :
Titre :
Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networks
Auteur(s) :
Biggs, Felix [Auteur]
Department of Computer science [University College of London] [UCL-CS]
The Inria London Programme [Inria-London]
Guedj, Benjamin [Auteur]
University College of London [London] [UCL]
Department of Computer science [University College of London] [UCL-CS]
Inria-CWI [Inria-CWI]
MOdel for Data Analysis and Learning [MODAL]
The Inria London Programme [Inria-London]
Department of Computer science [University College of London] [UCL-CS]
The Inria London Programme [Inria-London]
Guedj, Benjamin [Auteur]
![refId](/themes/Mirage2//images/idref.png)
University College of London [London] [UCL]
Department of Computer science [University College of London] [UCL-CS]
Inria-CWI [Inria-CWI]
MOdel for Data Analysis and Learning [MODAL]
The Inria London Programme [Inria-London]
Titre de la revue :
Entropy
Éditeur :
MDPI
Date de publication :
2021
ISSN :
1099-4300
Discipline(s) HAL :
Informatique [cs]/Apprentissage [cs.LG]
Statistiques [stat]/Machine Learning [stat.ML]
Statistiques [stat]/Machine Learning [stat.ML]
Résumé en anglais : [en]
We make three related contributions motivated by the challenge of training stochastic neural networks, particularly in a PAC-Bayesian setting: (1) we show how averaging over an ensemble of stochastic neural networks enables ...
Lire la suite >We make three related contributions motivated by the challenge of training stochastic neural networks, particularly in a PAC-Bayesian setting: (1) we show how averaging over an ensemble of stochastic neural networks enables a new class of \emph{partially-aggregated} estimators; (2) we show that these lead to provably lower-variance gradient estimates for non-differentiable signed-output networks; (3) we reformulate a PAC-Bayesian bound for these networks to derive a directly optimisable, differentiable objective and a generalisation guarantee, without using a surrogate loss or loosening the bound. This bound is twice as tight as that of Letarte et al. (2019) on a similar network type. We show empirically that these innovations make training easier and lead to competitive guarantees.Lire moins >
Lire la suite >We make three related contributions motivated by the challenge of training stochastic neural networks, particularly in a PAC-Bayesian setting: (1) we show how averaging over an ensemble of stochastic neural networks enables a new class of \emph{partially-aggregated} estimators; (2) we show that these lead to provably lower-variance gradient estimates for non-differentiable signed-output networks; (3) we reformulate a PAC-Bayesian bound for these networks to derive a directly optimisable, differentiable objective and a generalisation guarantee, without using a surrogate loss or loosening the bound. This bound is twice as tight as that of Letarte et al. (2019) on a similar network type. We show empirically that these innovations make training easier and lead to competitive guarantees.Lire moins >
Langue :
Anglais
Vulgarisation :
Non
Projet ANR :
Collections :
Source :
Fichiers
- document
- Accès libre
- Accéder au document
- 2006.12228.pdf
- Accès libre
- Accéder au document
- 2006.12228
- Accès libre
- Accéder au document