Title :

Generation and detection of manipulated multimodal audiovisual content: Advances, trends and open challenges

Author(s) :

Liz-López, Helena [Auteur correspondant]
Universidad Politécnica de Madrid [UPM]
Keita, Mamadou [Auteur]
Institut d’Électronique, de Microélectronique et de Nanotechnologie - UMR 8520 [IEMN]
COMmunications NUMériques - IEMN [COMNUM - IEMN]
Tahleb Ahmed, Abdelmalik [Auteur]
COMmunications NUMériques - IEMN [COMNUM - IEMN]
Institut d’Électronique, de Microélectronique et de Nanotechnologie - UMR 8520 [IEMN]
Hadid, Abdenour [Auteur]
Sorbonne University Abu Dhabi [SUAD]
Huertas-Tato, Javier [Auteur]
Universidad Politécnica de Madrid [UPM]
Camacho, David [Auteur]
Universidad Politécnica de Madrid [UPM]

Journal title :

Information Fusion

Pages :

102103

Publisher :

Elsevier

Publication date :

2024-03

ISSN :

1566-2535

English keyword(s) :

Multimedia data manipulation generation
Multimedia data forensics
Deep Learning
Video
Audio
Multimodal

HAL domain(s) :

Physique [physics]
Sciences de l'ingénieur [physics]

English abstract : [en]

Generative deep learning techniques have invaded the public discourse recently. Despite the advantages, the applications to disinformation are concerning as the counter-measures advance slowly. As the manipulation of ...
Show more >Generative deep learning techniques have invaded the public discourse recently. Despite the advantages, the applications to disinformation are concerning as the counter-measures advance slowly. As the manipulation of multimedia content becomes easier, faster, and more credible, developing effective forensics becomes invaluable. Other works have identified this need but neglect that disinformation is inherently multimodal. Overall in this survey, we exhaustively describe modern manipulation and forensic techniques from the lens of video, audio and their multimodal fusion. For manipulation techniques, we give a classification of the most commonly applied manipulations. Generative techniques can be exploited to generate datasets; we provide a list of current datasets useful for forensics. We have reviewed forensic techniques from 2018 to 2023, examined the usage of datasets, and given a comparative analysis of each modality. Finally, we give another comparison of end-to-end forensics tools for end-users. From our analysis clear trends are found with diffusion models, dataset granularity, explainability techniques, synchronisation improvements, and learning task diversity. We find a roadmap of deep challenges ahead, including multilinguality, multimodality, improving data quality (and variety), all in an adversarial ever-changing environment.Show less >

Language :

Anglais

Peer reviewed article :

Oui

Audience :

Internationale

Popular science :

Non

Collections :

• Institut d'Électronique, de Microélectronique et de Nanotechnologie (IEMN) - UMR 8520

Source :

Harvested from HAL