GAVEL: Grounded Caption Error Verification and Localization

2026-06-25

AI Summary

Existing vision-language models often generate descriptions inconsistent with input images and lack effective mechanisms for verifying and explaining such mismatches. This work introduces the GAVEL task, which unifies image-text consistency verification, natural language explanation generation, and fine-grained visual evidence localization into a single end-to-end learnable framework, accompanied by the first high-quality, human-annotated dataset for this purpose. The authors propose a supervised baseline model that leverages vision-language alignment and multi-task joint optimization, significantly outperforming strong closed-source models in both localization accuracy and explanation quality. These results underscore the challenge posed by the GAVEL task and demonstrate the effectiveness of the proposed approach.

Abstract

Vision-language models (VLMs) often produce hallucinated or inconsistent outputs, where text and images are not properly aligned. Addressing this issue requires not only detecting misalignment but also explaining the discrepancy and localizing its visual evidence. We introduce GAVEL (Grounded Caption Error Verification and Localization), a task that jointly addresses verification, explanation, and localization for image-text pairs. To support systematic evaluation, we also present a corresponding dataset and benchmark. We further train a supervised baseline on the human-annotated training split to assess whether GAVEL provides learnable supervision for these abilities. Experiments show that even strong closed-source models struggle on GAVEL, while the supervised baseline yields consistent improvements across grounding and explanation metrics.

Problem

• vision-language models
• hallucination
• image-text alignment
• error localization
• caption verification

Innovation

• vision-language alignment
• hallucination detection
• error localization
• multimodal explanation
• grounded captioning