Visual Reasoning with Multi-hop Feature Modulation
Document type :
Communication dans un congrès avec actes
Title :
Visual Reasoning with Multi-hop Feature Modulation
Author(s) :
Strub, Florian [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Université de Lille
Sequential Learning [SEQUEL]
Seurin, Mathieu [Auteur]
Université de Lille
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Sequential Learning [SEQUEL]
Perez, Ethan [Auteur]
Rice University [Houston]
de Vries, Harm [Auteur]
Department of Computer Science and Operations Research [Montreal]
Montreal Institute for Learning Algorithms [Montréal] [MILA]
Mary, Jérémie [Auteur]
Criteo [Paris]
Preux, Philippe [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Université de Lille
Sequential Learning [SEQUEL]
Courville, Aaron [Auteur]
Montreal Institute for Learning Algorithms [Montréal] [MILA]
CIFAR
Pietquin, Olivier [Auteur]
Google Inc
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Université de Lille
Sequential Learning [SEQUEL]
Seurin, Mathieu [Auteur]
Université de Lille
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Sequential Learning [SEQUEL]
Perez, Ethan [Auteur]
Rice University [Houston]
de Vries, Harm [Auteur]
Department of Computer Science and Operations Research [Montreal]
Montreal Institute for Learning Algorithms [Montréal] [MILA]
Mary, Jérémie [Auteur]
Criteo [Paris]
Preux, Philippe [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Université de Lille
Sequential Learning [SEQUEL]
Courville, Aaron [Auteur]
Montreal Institute for Learning Algorithms [Montréal] [MILA]
CIFAR
Pietquin, Olivier [Auteur]
Google Inc
Scientific editor(s) :
Vittorio Ferrari
Martial Hebert
Cristian Sminchisescu
Yair Weiss
Martial Hebert
Cristian Sminchisescu
Yair Weiss
Conference title :
ECCV 2018 - 15th European Conference on Computer Vision
City :
Munich
Country :
Allemagne
Start date of the conference :
2018-09-08
Journal title :
Proc. ECCV
Publisher :
Springer
Publication date :
2018-09-14
English keyword(s) :
Deep Learning
Computer Vision
Natural Language Understanding
Multi-modal Learning
Computer Vision
Natural Language Understanding
Multi-modal Learning
HAL domain(s) :
Informatique [cs]/Intelligence artificielle [cs.AI]
Informatique [cs]/Vision par ordinateur et reconnaissance de formes [cs.CV]
Informatique [cs]/Apprentissage [cs.LG]
Informatique [cs]/Réseau de neurones [cs.NE]
Informatique [cs]/Vision par ordinateur et reconnaissance de formes [cs.CV]
Informatique [cs]/Apprentissage [cs.LG]
Informatique [cs]/Réseau de neurones [cs.NE]
English abstract : [en]
Recent breakthroughs in computer vision and natural language processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue. For such tasks, one successful approach ...
Show more >Recent breakthroughs in computer vision and natural language processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue. For such tasks, one successful approach is to condition image-based convolutional network computation on language via Feature-wise Linear Modulation (FiLM) layers, i.e., per-channel scaling and shifting. We propose to generate the parameters of FiLM layers going up the hierarchy of a convolutional network in a multi-hop fashion rather than all at once, as in prior work. By alternating between attending to the language input and generating FiLM layer parameters, this approach is better able to scale to settings with longer input sequences such as dialogue. We demonstrate that multi-hop FiLM generation achieves state-of-the-art for the short input sequence task ReferIt-on-par with single-hop FiLM generation-while also significantly outperforming prior state-of-the-art and single-hop FiLM generation on the GuessWhat?! visual dialogue task.Show less >
Show more >Recent breakthroughs in computer vision and natural language processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue. For such tasks, one successful approach is to condition image-based convolutional network computation on language via Feature-wise Linear Modulation (FiLM) layers, i.e., per-channel scaling and shifting. We propose to generate the parameters of FiLM layers going up the hierarchy of a convolutional network in a multi-hop fashion rather than all at once, as in prior work. By alternating between attending to the language input and generating FiLM layer parameters, this approach is better able to scale to settings with longer input sequences such as dialogue. We demonstrate that multi-hop FiLM generation achieves state-of-the-art for the short input sequence task ReferIt-on-par with single-hop FiLM generation-while also significantly outperforming prior state-of-the-art and single-hop FiLM generation on the GuessWhat?! visual dialogue task.Show less >
Language :
Anglais
Peer reviewed article :
Oui
Audience :
Internationale
Popular science :
Non
European Project :
Collections :
Source :
Files
- https://hal.archives-ouvertes.fr/hal-01927811/document
- Open access
- Access the document
- http://arxiv.org/pdf/1808.04446
- Open access
- Access the document
- https://hal.archives-ouvertes.fr/hal-01927811/document
- Open access
- Access the document
- https://hal.archives-ouvertes.fr/hal-01927811/document
- Open access
- Access the document
- document
- Open access
- Access the document
- 1808.04446.pdf
- Open access
- Access the document
- 1808.04446
- Open access
- Access the document