MMGT: Multimodal Graph-based Transformer ...
Document type :
Communication dans un congrès avec actes
Title :
MMGT: Multimodal Graph-based Transformer for Pain Detection
Author(s) :
Feghoul, Kevin [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Lille Neurosciences & Cognition - U 1172 [LilNCog]
Santana Maia, Deise [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Daoudi, Mohamed [Auteur]
Ecole nationale supérieure Mines-Télécom Lille Douai [IMT Nord Europe]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Amad, Ali [Auteur]
Lille Neurosciences & Cognition - U 1172 [LilNCog]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Lille Neurosciences & Cognition - U 1172 [LilNCog]
Santana Maia, Deise [Auteur]

Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Daoudi, Mohamed [Auteur]
Ecole nationale supérieure Mines-Télécom Lille Douai [IMT Nord Europe]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Amad, Ali [Auteur]

Lille Neurosciences & Cognition - U 1172 [LilNCog]
Conference title :
31st European Signal Processing Conference (EUSIPCO 2023)
City :
Helsinki
Country :
Finlande
Start date of the conference :
2023-09-04
English keyword(s) :
Multimodal Learning
Transformer
Graph Convolutional Networks
Affective Computing
Transformer
Graph Convolutional Networks
Affective Computing
HAL domain(s) :
Informatique [cs]/Intelligence artificielle [cs.AI]
English abstract : [en]
Pain can be expressed from multiple modalities, such as facial expressions, physiological signals, and behaviors. For that reason, multimodal learning can greatly benefit automatic pain detection and, more generally, a ...
Show more >Pain can be expressed from multiple modalities, such as facial expressions, physiological signals, and behaviors. For that reason, multimodal learning can greatly benefit automatic pain detection and, more generally, a variety of tasks in the field of affective computing. In this context, as one of our main contributions, we leverage the multimodal interaction among the intermediate modality representations, which are rarely exploited in existing works. In order to capture the relationships between multiple modalities, we propose the Multimodal Graph-based Transformer (MMGT), in which unimodality feature extraction is performed using Transformers and then fused using a Graph Convolutional Network (GCN). We evaluated MMGT on the BP4D+ dataset, and the results demonstrate the efficiency of our fusion framework for the task of pain detection, which outperformed all the existing approaches under multimodal settings. Our best results were obtained using 2D facial landmarks, action units, and physiological data, on which we achieved 94.95% and 94.91% of accuracy and F1-score, respectively.Show less >
Show more >Pain can be expressed from multiple modalities, such as facial expressions, physiological signals, and behaviors. For that reason, multimodal learning can greatly benefit automatic pain detection and, more generally, a variety of tasks in the field of affective computing. In this context, as one of our main contributions, we leverage the multimodal interaction among the intermediate modality representations, which are rarely exploited in existing works. In order to capture the relationships between multiple modalities, we propose the Multimodal Graph-based Transformer (MMGT), in which unimodality feature extraction is performed using Transformers and then fused using a Graph Convolutional Network (GCN). We evaluated MMGT on the BP4D+ dataset, and the results demonstrate the efficiency of our fusion framework for the task of pain detection, which outperformed all the existing approaches under multimodal settings. Our best results were obtained using 2D facial landmarks, action units, and physiological data, on which we achieved 94.95% and 94.91% of accuracy and F1-score, respectively.Show less >
Language :
Anglais
Peer reviewed article :
Oui
Audience :
Internationale
Popular science :
Non
Collections :
Source :
Files
- document
- Open access
- Access the document
- 2023_EUSIPCO_MMGT_final.pdf
- Open access
- Access the document