DualFact+: A Multimodal Fact Verification Framework for Procedural Video Understanding

2026-04-28

AI Summary

Existing evaluation methods for procedural video captioning struggle to accurately assess factual correctness, often misidentifying hallucinations due to insufficient attention to character consistency and contextual details. This work proposes DualFact, a two-tier multimodal fact verification framework that decomposes facts into a conceptual layer (e.g., actions, tools, and semantic roles) and a contextual layer (their concrete realizations in the video), verifying each using either textual (DualFact-T) or visual (DualFact-V) evidence. DualFact incorporates implicit argument augmentation (VIA) and contrastive fact construction to effectively distinguish between these two fact types, yielding an interpretable evaluation protocol that aligns closely with human judgment. Experiments on YouCook3-Fact and CraftBench-Fact demonstrate significant improvements over existing metrics and reveal that caption-only evaluations tend to overestimate hallucination rates.

Abstract

We introduce DualFact, a dual-layer, multimodal factuality evaluation framework for procedural video captioning. DualFact separates factual correctness into conceptual facts, capturing abstract semantic roles (e.g., Action, Ingredient, Tool, Location), and contextual facts, capturing their grounded predicate-argument realizations in video. To support complete and role-consistent evaluation, DualFact incorporates implicit argument augmentation (VIA) and contrastive fact sets. We instantiate DualFact in two modes: DualFact-T, which verifies facts against textual evidence, and DualFact-V, which verifies facts against video-grounded visual evidence. Experiments on YouCook3-Fact and CraftBench-Fact show that state-of-the-art multimodal language models produce fluent but often factually incomplete captions, with systematic omissions and role-level inconsistencies. DualFact correlates more strongly with human factuality judgments than standard metrics, particularly for contextual facts, and reveals that caption-only evaluation overestimates hallucinations compared to video-grounded verification. Overall, DualFact offers an interpretable and human-aligned evaluation protocol that highlights persistent challenges in multimodal factual grounding, extending beyond surface-level fluency.

Problem

• factuality
• procedural video understanding
• multimodal evaluation
• hallucination
• video captioning

Innovation

• multimodal fact verification
• procedural video understanding
• implicit argument augmentation
• dual-layer evaluation
• video-grounded factuality