FastV-RAG: Towards Fast and Fine-Grained Video QA with Retrieval-Augmented Generation

2026-01-04

arXiv.org

AI Summary

This work proposes VideoSpeculateRAG, a novel framework that introduces speculative decoding into multimodal retrieval-augmented generation (RAG) for video question answering. Addressing the challenges of low inference efficiency and insufficient answer accuracy in existing RAG-based approaches when incorporating external knowledge, VideoSpeculateRAG employs a lightweight draft model and a heavyweight verification model that collaboratively generate responses. The framework further integrates similarity-based retrieval filtering and entity alignment mechanisms to mitigate retrieval noise and misidentification issues. Experimental results demonstrate that VideoSpeculateRAG achieves approximately 2× inference speedup while maintaining comparable or higher answer accuracy, significantly enhancing both the efficiency and reliability of video question-answering systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Vision-Language Models (VLMs) excel at visual reasoning but still struggle with integrating external knowledge. Retrieval-Augmented Generation (RAG) is a promising solution, but current methods remain inefficient and often fail to maintain high answer quality. To address these challenges, we propose VideoSpeculateRAG, an efficient VLM-based RAG framework built on two key ideas. First, we introduce a speculative decoding pipeline: a lightweight draft model quickly generates multiple answer candidates, which are then verified and refined by a more accurate heavyweight model, substantially reducing inference latency without sacrificing correctness. Second, we identify a major source of error - incorrect entity recognition in retrieved knowledge - and mitigate it with a simple yet effective similarity-based filtering strategy that improves entity alignment and boosts overall answer accuracy. Experiments demonstrate that VideoSpeculateRAG achieves comparable or higher accuracy than standard RAG approaches while accelerating inference by approximately 2x. Our framework highlights the potential of combining speculative decoding with retrieval-augmented reasoning to enhance efficiency and reliability in complex, knowledge-intensive multimodal tasks.

Problem

• Video QA
• Retrieval-Augmented Generation
• Vision-Language Models
• Inference Efficiency
• External Knowledge Integration

Innovation

• Speculative Decoding
• Retrieval-Augmented Generation
• Entity Alignment
• Video Question Answering
• Vision-Language Models