Learning via Wasserstein-Based High Probability Generalisation Bounds

Viallard, Paul; Haddouche, Maxime; Şimşekli, Umut; Guedj, Benjamin

Type de document :

Communication dans un congrès avec actes

DOI :

10.48550/arXiv.2306.04375

Titre :

Learning via Wasserstein-Based High Probability Generalisation Bounds

Auteur(s) :

Viallard, Paul [Auteur]
Statistical Machine Learning and Parsimony [SIERRA]
Haddouche, Maxime [Auteur]
The Inria London Programme [Inria-London]
MOdel for Data Analysis and Learning [MODAL]
Şimşekli, Umut [Auteur]
Statistical Machine Learning and Parsimony [SIERRA]
Guedj, Benjamin [Auteur]

University College of London [London] [UCL]
Department of Computer science [University College of London] [UCL-CS]
The Inria London Programme [Inria-London]
MOdel for Data Analysis and Learning [MODAL]

Titre de la manifestation scientifique :

NeurIPS 2023 - Thirty-seventh Conference on Neural Information Processing Systems

Ville :

New Orleans

Pays :

Etats-Unis d'Amérique

Date de début de la manifestation scientifique :

2023-12-10

Date de publication :

2023-06-07

Discipline(s) HAL :

Statistiques [stat]/Machine Learning [stat.ML]

Résumé en anglais : [en]

Minimising upper bounds on the population risk or the generalisation gap has been widely used in structural risk minimisation (SRM) -- this is in particular at the core of PAC-Bayesian learning. Despite its successes and ...
Lire la suite >Minimising upper bounds on the population risk or the generalisation gap has been widely used in structural risk minimisation (SRM) -- this is in particular at the core of PAC-Bayesian learning. Despite its successes and unfailing surge of interest in recent years, a limitation of the PAC-Bayesian framework is that most bounds involve a Kullback-Leibler (KL) divergence term (or its variations), which might exhibit erratic behavior and fail to capture the underlying geometric structure of the learning problem -- hence restricting its use in practical applications. As a remedy, recent studies have attempted to replace the KL divergence in the PAC-Bayesian bounds with the Wasserstein distance. Even though these bounds alleviated the aforementioned issues to a certain extent, they either hold in expectation, are for bounded losses, or are nontrivial to minimize in an SRM framework. In this work, we contribute to this line of research and prove novel Wasserstein distance-based PAC-Bayesian generalisation bounds for both batch learning with independent and identically distributed (i.i.d.) data, and online learning with potentially non-i.i.d. data. Contrary to previous art, our bounds are stronger in the sense that (i) they hold with high probability, (ii) they apply to unbounded (potentially heavy-tailed) losses, and (iii) they lead to optimizable training objectives that can be used in SRM. As a result we derive novel Wasserstein-based PAC-Bayesian learning algorithms and we illustrate their empirical advantage on a variety of experiments.Lire moins >

Langue :

Anglais

Comité de lecture :

Oui

Audience :

Internationale

Vulgarisation :

Non

Projet ANR :

PaRis Artificial Intelligence Research InstitutE
Apprentissage PAC-bayésien agnostique
Une Perspective PAC-Bayésienne de l'Apprentissage de Représentations

Collections :