Self-Supervised Contrastive Learning with Mamba-2 State Space Models for Audio-Visual Deepfake Detection

Abstract

With the rapid proliferation of generative adversarial networks (GANs) and diffusion models, deepfake videos have achieved a level of fidelity that challenges the limits of human visual perception, posing severe threats to information security and public trust. Existing detection methodologies mostly rely on Convolutional Neural Networks (CNNs) combined with Recurrent Neural Networks (RNNs) or Transformers to extract spatial-temporal features. However, Transformers scale quadratically with long sequences. Additionally, traditional fusion methods miss subtle audio-visual inconsistencies. This paper proposes a novel, lightweight multi-modal detection framework that integrates Mamba-2 State Space Models (SSM) with Self-supervised Contrastive Learning (SSCL). Specifically, we leverage the linear complexity and the selective scan mechanism of Mamba-2 to efficiently model long-range temporal dependencies. Furthermore, a dual-branch contrastive loss is introduced to explicitly align the physical synchronization patterns between lip movements and speech phonemes. This forces the model to learn invariant forensic features rather than overfitting to specific visual artifacts. Experiments demonstrate that our proposed method achieves superior performance: an AUC of 0.976 on FaceForensics++ and 0.962 on FakeAVCeleb.

Keywords: Deepfake detection, mamba-2, state space models, multi-modal learning, contrastive learning