Textual Acoustic Grounding for Generalizable LLM-Based Deepfake Voice Detection

2026-08-31

AI Summary

为解决深伪音频检测跨域泛化差的问题，通过结合Qwen语言模型与音频编码器，并引入跨模态提示策略，显著提升了检测性能。

Abstract

Deepfake voice detection suffers from poor generalization across unseen domains. While Audio Large Language Models (ALLMs) show promise, the modality gap between continuous audio embeddings which capture the subtle acoustic details necessary for deepfake detection and the semantic space of LLMs remains a critical, underexplored bottleneck. We address this by benchmarking diverse audio encoders integrated with Qwen LLMs (0.5B to 7B parameters). First, we demonstrate that fine-tuning the LLM alone risks out-of-domain overfitting, making a frozen LLM a stronger, resource-efficient baseline. Second, to explicitly bridge the modality gap, we introduce a cross-modal prompting strategy that injects linguistic-knowledge-driven acoustic features (via openSMILE) as structured text tokens. This explicit textual grounding not only enhances the frozen baseline but also makes LLM fine-tuning more effective. Ultimately, our approach demonstrates state-of-the-art resilience on the out-of-domain ITW and MLAAD benchmarks, yielding over \textbf{16.2\%} absolute improvement in Macro-F1 over existing ALLM baselines while maintaining competitive in-domain performance. All models reported in this work are \href{https://huggingface.co/01Yassine/AudioLLM-Deepfake-Detection}{publicly available}.

Problem

• Deepfake Voice Detection
• Generalization
• Modality Gap
• Audio Large Language Models
• Continuous Audio Embeddings

Innovation

• cross-modal prompting
• linguistic-knowledge-driven acoustic features
• modality gap
• frozen LLM