A Knowledge Noise Mitigation Framework for Knowledge-based Visual Question Answering

2025-09-11

AI Summary

In KB-VQA, external knowledge retrieval often introduces substantial redundancy and noise, degrading answer accuracy. To address this, we propose a training-free, knowledge-focusing framework that enhances the utilization efficiency of highly relevant knowledge. First, it generates low-noise queries to improve knowledge retrieval quality. Second, it leverages large language models for prompt-based extraction of salient knowledge snippets. Finally, it employs a confidence-aware dynamic ensemble mechanism to selectively fuse retrieved knowledge. Crucially, the framework avoids end-to-end fine-tuning, ensuring lightweight deployment and strong generalization across diverse KB-VQA settings. Evaluated on multiple mainstream KB-VQA benchmarks, our method achieves state-of-the-art performance in both answer accuracy and noise suppression, demonstrating the effectiveness of explicit knowledge denoising and focus strategies.

Technology Category

• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Natural Language Processing: Question Answering

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Knowledge-based visual question answering (KB-VQA) requires a model to understand images and utilize external knowledge to provide accurate answers. Existing approaches often directly augment models with retrieved information from knowledge sources while ignoring substantial knowledge redundancy, which introduces noise into the answering process. To address this, we propose a training-free framework with knowledge focusing for KB-VQA, that mitigates the impact of noise by enhancing knowledge relevance and reducing redundancy. First, for knowledge retrieval, our framework concludes essential parts from the image-question pairs, creating low-noise queries that enhance the retrieval of highly relevant knowledge. Considering that redundancy still persists in the retrieved knowledge, we then prompt large models to identify and extract answer-beneficial segments from knowledge. In addition, we introduce a selective knowledge integration strategy, allowing the model to incorporate knowledge only when it lacks confidence in answering the question, thereby mitigating the influence of redundant information. Our framework enables the acquisition of accurate and critical knowledge, and extensive experiments demonstrate that it outperforms state-of-the-art methods.

Problem

• Mitigating knowledge noise in visual question answering
• Reducing redundancy in retrieved external knowledge
• Enhancing knowledge relevance for accurate answers

Innovation

• Training-free framework with knowledge focusing
• Low-noise queries for relevant knowledge retrieval
• Selective integration only when confidence lacking