Title :

Accelerated NAS via pretrained ensembles and multi-fidelity Bayesian Optimization

Author(s) :

Ouertatani, Houssem [Auteur]
IRT SystemX
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Maxim, Cristian [Auteur]
IRT SystemX
Niar, Smail [Auteur]
Université Polytechnique Hauts-de-France [UPHF]
Laboratoire d'Automatique, de Mécanique et d'Informatique industrielles et Humaines - UMR 8201 [LAMIH]
Talbi, El-Ghazali [Auteur]
Optimisation de grande taille et calcul large échelle [BONUS]
Université de Lille
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Centre Inria de l'Université de Lille

Conference title :

33rd International Conference on Artificial Neural Networks (ICANN)

City :

Lugano

Country :

Suisse

Start date of the conference :

2024-09-17

Book title :

Conference proceedings are published by Springer in Lecture Notes in Computer Science

English keyword(s) :

Neural Architecture Search
Deep Ensembles
Multi-fidelity BO

HAL domain(s) :

Informatique [cs]/Intelligence artificielle [cs.AI]
Mathématiques [math]/Optimisation et contrôle [math.OC]

English abstract : [en]

Bayesian optimization (BO) is a black-box search method particularly valued for its sample efficiency. It is especially effective when evaluations are very costly, such as in hyperparameter optimization or Neural Architecture ...
Show more >Bayesian optimization (BO) is a black-box search method particularly valued for its sample efficiency. It is especially effective when evaluations are very costly, such as in hyperparameter optimization or Neural Architecture Search (NAS). In this work, we design a fast NAS method based on BO. While Gaussian Processes underpin most BO approaches, we instead use deep ensembles. This allows us to construct a unified and improved representation, leveraging pretraining metrics and multiple evaluation fidelities, to accelerate the search. More specifically, we use a simultaneous pretraining scheme where multiple metrics are estimated concurrently. Consequently, a more general representation is obtained. A novel multi-fidelity approach is proposed, where the unified representation is improved both by high and low quality evaluations. These additions significantly accelerate the search time, finding the optimum on NAS-Bench-201 in an equivalent time and cost to performing as few as 50 to 80 evaluations. The accelerated search time translates to reduced costs, in terms of computing resources and energy consumption. As a result, applying this NAS method to real-world use cases becomes more practical and not prohibitively expensive. We demonstrate the effectiveness and generality of our approach on a custom search space. Based on the MOAT architecture, we design a search space of CNN-ViT hybrid networks. The search method yields a better-performing architecture than the baseline in only 70 evaluations.Show less >

Language :

Anglais

Peer reviewed article :

Oui

Audience :

Internationale

Popular science :

Non

ANR Project :

Methods and Algorithms for Exascale

Collections :

• Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189

Source :

Harvested from HAL