Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

2026-04-12

AI Summary

This work addresses the significant performance degradation of existing audio-visual question answering methods under missing modalities, a problem exacerbated by generative imputation strategies that often introduce hallucinations and reasoning biases. To overcome these limitations, the authors propose R²ScP, a novel retrieval-based framework for missing modality recovery. R²ScP leverages a unified semantic embedding space to enable cross-modal knowledge retrieval and incorporates a context-aware adaptive purification mechanism to filter out semantic noise. A two-stage training strategy is employed to preserve modality-specific information while enhancing reasoning robustness. Experimental results demonstrate that R²ScP substantially outperforms current approaches in incomplete-modality scenarios, effectively improving system stability and question-answering accuracy.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent Audio-Visual Question Answering (AVQA) methods have advanced significantly. However, most AVQA methods lack effective mechanisms for handling missing modalities, suffering from severe performance degradation in real-world scenarios with data interruptions. Furthermore, prevailing methods for handling missing modalities predominantly rely on generative imputation to synthesize missing features. While partially effective, these methods tend to capture inter-modal commonalities but struggle to acquire unique, modality-specific knowledge within the missing data, leading to hallucinations and compromised reasoning accuracy. To tackle these challenges, we propose R$^{2}$ScP, a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery. Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge. To maximize semantic restoration, we introduce a context-aware adaptive purification mechanism that eliminates latent semantic noise within the retrieved data. Additionally, we employ a two-stage training strategy to explicitly model the semantic relationships between knowledge from different sources. Extensive experiments demonstrate that R$^{2}$ScP significantly improves AVQA and enhances robustness in modal-incomplete scenarios.

Problem

• Audio-Visual Question Answering
• Missing Modality
• Modality Imputation
• Semantic Consistency
• Robustness

Innovation

• retrieval-based recovery
• semantic-consistent purification
• incomplete AVQA
• cross-modal retrieval
• adaptive denoising