Perturbed Model Validation: A New Framework to Validate Model Relevance

Zhang, Jie; Barr, Earl; Guedj, Benjamin; Harman, Mark; Shawe-Taylor, John

Document type :

Pré-publication ou Document de travail

Title :

Perturbed Model Validation: A New Framework to Validate Model Relevance

Author(s) :

Zhang, Jie [Auteur]
University College of London [London] [UCL]
Department of Computer science [University College of London] [UCL-CS]
Barr, Earl [Auteur]
University College of London [London] [UCL]
Guedj, Benjamin [Auteur]

Department of Computer science [University College of London] [UCL-CS]
University College of London [London] [UCL]
MOdel for Data Analysis and Learning [MODAL]
The Inria London Programme [Inria-London]
Harman, Mark [Auteur]
University College of London [London] [UCL]
Facebook
Shawe-Taylor, John [Auteur]
University College of London [London] [UCL]
Department of Computer science [University College of London] [UCL-CS]
The Inria London Programme [Inria-London]

HAL domain(s) :

Informatique [cs]/Apprentissage [cs.LG]
Statistiques [stat]/Machine Learning [stat.ML]

English abstract : [en]

This paper introduces Perturbed Model Validation (PMV), a new technique to validate model relevance and detect overfitting or underfitting. PMV operates by injecting noise to the training data, re-training the model against ...
Show more >This paper introduces Perturbed Model Validation (PMV), a new technique to validate model relevance and detect overfitting or underfitting. PMV operates by injecting noise to the training data, re-training the model against the perturbed data, then using the training accuracy decrease rate to assess model relevance. A larger decrease rate indicates better concept-hypothesis fit. We realise PMV by perturbing labels to inject noise, and evaluate PMV on four real-world datasets (breast cancer, adult, connect-4, and MNIST) and nine synthetic datasets in the classification setting. The results reveal that PMV selects models more precisely and in a more stable way than cross-validation, and effectively detects both overfitting and underfitting.Show less >

Language :

Anglais

Collections :

Laboratoire Paul Painlevé - UMR 8524

Source :

Harvested from HAL

Files

document
Open access
Access the document

main.pdf
Open access
Access the document

Perturbed Model Validation: A New Framework ... BibTeX CSV Excel RIS

Files

Perturbed Model Validation: A New Framework ...

BibTeX

CSV

Excel

RIS