Title :

Visual Reasoning with Multi-hop Feature Modulation

Author(s) :

Strub, Florian [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Université de Lille
Sequential Learning [SEQUEL]
Seurin, Mathieu [Auteur]
Université de Lille
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Sequential Learning [SEQUEL]
Perez, Ethan [Auteur]
Rice University [Houston]
de Vries, Harm [Auteur]
Department of Computer Science and Operations Research [Montreal]
Montreal Institute for Learning Algorithms [Montréal] [MILA]
Mary, Jérémie [Auteur]
Criteo [Paris]
Preux, Philippe [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Université de Lille
Sequential Learning [SEQUEL]
Courville, Aaron [Auteur]
Montreal Institute for Learning Algorithms [Montréal] [MILA]
CIFAR
Pietquin, Olivier [Auteur]
Google Inc

Scientific editor(s) :

Vittorio Ferrari
Martial Hebert
Cristian Sminchisescu
Yair Weiss

Conference title :

ECCV 2018 - 15th European Conference on Computer Vision

City :

Munich

Country :

Allemagne

Start date of the conference :

2018-09-08

Journal title :

Proc. ECCV

Publisher :

Springer

Publication date :

2018-09-14

English keyword(s) :

Deep Learning
Computer Vision
Natural Language Understanding
Multi-modal Learning

HAL domain(s) :

Informatique [cs]/Intelligence artificielle [cs.AI]
Informatique [cs]/Vision par ordinateur et reconnaissance de formes [cs.CV]
Informatique [cs]/Apprentissage [cs.LG]
Informatique [cs]/Réseau de neurones [cs.NE]

English abstract : [en]

Recent breakthroughs in computer vision and natural language processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue. For such tasks, one successful approach ...
Show more >Recent breakthroughs in computer vision and natural language processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue. For such tasks, one successful approach is to condition image-based convolutional network computation on language via Feature-wise Linear Modulation (FiLM) layers, i.e., per-channel scaling and shifting. We propose to generate the parameters of FiLM layers going up the hierarchy of a convolutional network in a multi-hop fashion rather than all at once, as in prior work. By alternating between attending to the language input and generating FiLM layer parameters, this approach is better able to scale to settings with longer input sequences such as dialogue. We demonstrate that multi-hop FiLM generation achieves state-of-the-art for the short input sequence task ReferIt-on-par with single-hop FiLM generation-while also significantly outperforming prior state-of-the-art and single-hop FiLM generation on the GuessWhat?! visual dialogue task.Show less >

Language :

Anglais

Peer reviewed article :

Oui

Audience :

Internationale

Popular science :

Non

European Project :

Child-Robot Communication and Collaboration: Edutainment, Behavioural Modelling and Cognitive Development in Typically Developing and Autistic Spectrum Children
CHIST-ERA project "Interactive Grounded Language Understanding"

Collections :

• Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189

Source :

Harvested from HAL