Differentially Private Coordinate Descent for Composite Empirical Risk Minimization

Mangold, Paul; Bellet, Aurelien; Salmon, Joseph; Tommasi, Marc

Document type :

Pré-publication ou Document de travail

Permalink :

http://hdl.handle.net/20.500.12210/57851

Title :

Differentially Private Coordinate Descent for Composite Empirical Risk Minimization

Author(s) :

Mangold, Paul [Auteur]
Machine Learning in Information Networks [MAGNET]
Bellet, Aurelien [Auteur]

Machine Learning in Information Networks [MAGNET]
Salmon, Joseph [Auteur]

Université de Montpellier [UM]
Institut Montpelliérain Alexander Grothendieck [IMAG]
Tommasi, Marc [Auteur]

Machine Learning in Information Networks [MAGNET]

HAL domain(s) :

Informatique [cs]/Apprentissage [cs.LG]
Statistiques [stat]/Machine Learning [stat.ML]

English abstract : [en]

Machine learning models can leak information about the data used to train them. Differentially Private (DP) variants of optimization algorithms like Stochastic Gradient Descent (DP-SGD) have been designed to mitigate this, ...
Show more >Machine learning models can leak information about the data used to train them. Differentially Private (DP) variants of optimization algorithms like Stochastic Gradient Descent (DP-SGD) have been designed to mitigate this, inducing a trade-off between privacy and utility. In this paper, we propose a new method for composite Differentially Private Empirical Risk Minimization (DP-ERM): Differentially Private proximal Coordinate Descent (DP-CD). We analyze its utility through a novel theoretical analysis of inexact coordinate descent, and highlight some regimes where DP-CD outperforms DP-SGD, thanks to the possibility of using larger step sizes. We also prove new lower bounds for composite DP-ERM under coordinate-wise regularity assumptions, that are, in some settings, nearly matched by our algorithm. In practical implementations, the coordinate-wise nature of DP-CD updates demands special care in choosing the clipping thresholds used to bound individual contributions to the gradients. A natural parameterization of these thresholds emerges from our theory, limiting the addition of unnecessarily large noise without requiring coordinatewise hyperparameter tuning or extra computational cost.Show less >

Language :

Anglais

Collections :

Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189

Source :

Harvested from HAL

Submission date :

2021-11-17T02:01:42Z

Files

https://hal.inria.fr/hal-03424974/document
Open access
Access the document

Differentially Private Coordinate Descent ... BibTeX CSV Excel RIS

Files

Differentially Private Coordinate Descent ...

BibTeX

CSV

Excel

RIS