Differentially Private Coordinate Descent for Composite Empirical Risk Minimization

Mangold, Paul; Bellet, Aurelien; Salmon, Joseph; Tommasi, Marc

Type de document :

Communication dans un congrès avec actes

Titre :

Differentially Private Coordinate Descent for Composite Empirical Risk Minimization

Auteur(s) :

Mangold, Paul [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Machine Learning in Information Networks [MAGNET]
Bellet, Aurelien [Auteur]

Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Machine Learning in Information Networks [MAGNET]
Salmon, Joseph [Auteur]
Scientific Data Management [ZENITH]
Institut universitaire de France [IUF]
Institut Montpelliérain Alexander Grothendieck [IMAG]
Tommasi, Marc [Auteur]

Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Machine Learning in Information Networks [MAGNET]

Éditeur(s) ou directeur(s) scientifique(s) :

Kamalika Chaudhuri
Stefanie Jegelka
Le Song
Csaba Szepesvari
Gang Niu
Sivan Sabato

Titre de la manifestation scientifique :

ICML 2022 - 39th International Conference on Machine Learning

Ville :

Baltimore

Pays :

Etats-Unis d'Amérique

Date de début de la manifestation scientifique :

2022-07-17

Titre de l’ouvrage :

Proceedings of the 39th International Conference on Machine Learning

Éditeur :

PMLR

Date de publication :

2022

Discipline(s) HAL :

Informatique [cs]/Apprentissage [cs.LG]
Statistiques [stat]/Machine Learning [stat.ML]

Résumé en anglais : [en]

Machine learning models can leak information about the data used to train them. To mitigate this issue, Differentially Private (DP) variants of optimization algorithms like Stochastic Gradient Descent (DP-SGD) have been ...
Lire la suite >Machine learning models can leak information about the data used to train them. To mitigate this issue, Differentially Private (DP) variants of optimization algorithms like Stochastic Gradient Descent (DP-SGD) have been designed to trade-off utility for privacy in Empirical Risk Minimization (ERM) problems. In this paper, we propose Differentially Private proximal Coordinate Descent (DP-CD), a new method to solve composite DP-ERM problems. We derive utility guarantees through a novel theoretical analysis of inexact coordinate descent. Our results show that, thanks to larger step sizes, DP-CD can exploit imbalance in gradient coordinates to outperform DP-SGD. We also prove new lower bounds for composite DP-ERM under coordinate-wise regularity assumptions, that are nearly matched by DP-CD. For practical implementations, we propose to clip gradients using coordinate-wise thresholds that emerge from our theory, avoiding costly hyperparameter tuning. Experiments on real and synthetic data support our results, and show that DP-CD compares favorably with DP-SGD.Lire moins >

Langue :

Anglais

Comité de lecture :

Oui

Audience :

Internationale

Vulgarisation :

Non

Projet ANR :

Apprentissage automatique décentralisé et préservant la vie privée
Apprentissage automatique et optimisation coopératifs.

Collections :