Titre :

Modulating early visual processing by language

Auteur(s) :

de Vries, Harm [Auteur]
Department of Computer Science and Operations Research [Montreal]
Strub, Florian [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Sequential Learning [SEQUEL]
Mary, Jérémie [Auteur]
Sequential Learning [SEQUEL]
Larochelle, Hugo [Auteur]
Google Brain
Pietquin, Olivier [Auteur]
DeepMind [London]
Courville, Aaron [Auteur]
Department of Computer Science and Operations Research [Montreal]

Titre de la manifestation scientifique :

NIPS 2017 - Conference on Neural Information Processing Systems

Ville :

Long Beach

Pays :

Etats-Unis d'Amérique

Date de début de la manifestation scientifique :

2017-12-04

Discipline(s) HAL :

Informatique [cs]/Réseau de neurones [cs.NE]
Informatique [cs]/Intelligence artificielle [cs.AI]
Informatique [cs]/Vision par ordinateur et reconnaissance de formes [cs.CV]

Résumé en anglais : [en]

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, ...
Lire la suite >It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic inputs are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the entire visual processing by a linguistic input. Specifically, we introduce Conditional Batch Normalization (CBN) as an efficient mechanism to modulate convolutional feature maps by a linguistic embedding. We apply CBN to a pre-trained Residual Network (ResNet), leading to the MODulatEd ResNet (MODERN) architecture, and show that this significantly improves strong baselines on two visual question answering tasks. Our ablation study confirms that modulating from the early stages of the visual processing is beneficial.Lire moins >

Langue :

Anglais

Comité de lecture :

Oui

Audience :

Internationale

Vulgarisation :

Non

Collections :

• Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189

Source :

Harvested from HAL