ReflectCAP: Detailed Image Captioning with Reflective Memory

2026-04-14

AI Summary

Existing image captioning methods struggle to simultaneously achieve factual accuracy and fine-grained coverage. This work proposes Reflective Note-Guided Captioning, a novel framework that introduces, for the first time, a structured reflective note mechanism. By leveraging multi-agent analysis of hallucination and omission patterns in large vision-language models—such as GPT-4.1, Qwen, and InternVL—the method distills structured reflective notes that guide caption generation during inference. Without increasing model scale, the approach achieves a Pareto-optimal trade-off between factuality and coverage while reducing computational overhead by 21–36%. It substantially outperforms strong baselines on the CapArena-Auto benchmark and demonstrates both high-quality outputs and efficiency in real-world scenarios.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Detailed image captioning demands both factual grounding and fine-grained coverage, yet existing methods have struggled to achieve them simultaneously. We address this tension with Reflective Note-Guided Captioning (ReflectCAP), where a multi-agent pipeline analyzes what the target large vision-language model (LVLM) consistently hallucinates and what it systematically overlooks, distilling these patterns into reusable guidelines called Structured Reflection Notes. At inference time, these notes steer the captioning model along both axes -- what to avoid and what to attend to -- yielding detailed captions that jointly improve factuality and coverage. Applying this method to 8 LVLMs spanning the GPT-4.1 family, Qwen series, and InternVL variants, ReflectCAP reaches the Pareto frontier of the trade-off between factuality and coverage, and delivers substantial gains on CapArena-Auto, where generated captions are judged head-to-head against strong reference models. Moreover, ReflectCAP offers a more favorable trade-off between caption quality and compute cost than model scaling or existing multi-agent pipelines, which incur 21--36\% greater overhead. This makes high-quality detailed captioning viable under real-world cost and latency constraints.

Problem

• detailed image captioning
• factuality
• coverage
• hallucination
• fine-grained description

Innovation

• Reflective Memory
• Structured Reflection Notes
• Detailed Image Captioning
• Multi-agent Pipeline
• Factuality-Coverage Trade-off