CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework

2026-06-16

AI Summary

This work addresses the hallucination problem in vision-language models caused by insufficient evidential support. To this end, the authors propose a modular agent framework grounded in retrieval-augmented generation (RAG) with a reflective mechanism, featuring a closed-loop five-stage pipeline—extraction, retrieval, reasoning, citation injection, and verification—that enables evidence-driven, interpretable reasoning and triggers targeted re-retrieval upon detecting unsupported claims. The study introduces a comprehensive evaluation suite comprising 23 stage-specific metrics, centered on the novel composite CaVeScore. Without modifying model architectures or prompts, the method achieves 87.1% accuracy (56.6% CaVeScore) on ScienceQA and 55.2% accuracy (35.7% CaVeScore) across 30 subjects in MMMU, substantially improving both reasoning traceability and correctness.

Abstract

Vision-Language Models (VLMs) remain prone to hallucinations, producing fluent but visually unfaithful outputs. Existing chain-of-thought and retrieval-augmented methods only partially address this, as they neither enforce step-level citation grounding nor route verification failures back to retrieval for correction. We present CaVe-VLM-CoT, a modular reflection-based agentic-RAG framework that enforces evidence-grounded reasoning through a five-stage closed-loop pipeline: Extractor, Retriever, Solver, Citation Injector, and Verifier, in which detected ungrounded claims trigger structured feedback to the Extractor for targeted re-retrieval. Since no existing framework jointly measures retrieval quality, step-wise citation faithfulness, and cross-modal grounding, we propose a suite of 23 component-wise metrics across all stages, anchored by CaVeScore, a composite metric weighting accuracy, citation precision and recall, attribution, and evidence grounding. Without any architectural or prompt modifications, CaVe-VLM-CoT achieves 87.1\% accuracy and 56.6\% CaVeScore on ScienceQA , and 55.2\% accuracy and 35.7\% CaVeScore on MMMU (30 subjects).

Problem

• hallucination
• vision-language models
• evidence grounding
• retrieval-augmented generation
• citation faithfulness

Innovation

• evidence-grounded reasoning
• agentic-RAG
• closed-loop retrieval
• citation faithfulness
• CaVeScore