ReCap: Lightweight Referential Grounding for Coherent Story Visualization

2026-04-20

AI Summary

This work addresses the challenge of maintaining cross-frame consistency in character identity, spatial layout, and visual style for story visualization. The authors propose ReCap, a lightweight framework that achieves effective consistency modeling without modifying the underlying diffusion model. ReCap innovatively leverages pronouns as visual anchors and introduces a conditional frame referencing mechanism (CORE module) that activates cross-frame conditioning only when necessary. It further incorporates semantic drift correction (SemDrift) to align DINOv3 visual embeddings, adding merely 149K parameters and incurring no additional inference overhead. Evaluated on the FlintstonesSV and PororoSV benchmarks, ReCap surpasses the previous state-of-the-art by 2.63% and 5.65% in character accuracy, respectively, and demonstrates successful extension to real-world cinematic portrait narrative scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Consent frameworks and practices on the web
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Story Visualization aims to generate a sequence of images that faithfully depicts a textual narrative that preserve character identity, spatial configuration, and stylistic coherence as the narratives unfold. Maintaining such cross-frame consistency has traditionally relied on explicit memory banks, architectural expansion, or auxiliary language models, resulting in substantial parameter growth and inference overhead. We introduce ReCap, a lightweight consistency framework that improves character stability and visual fidelity without modifying the base diffusion backbone. ReCap's CORE (COnditional frame REferencing) module treats anaphors, in our case pronouns, as visual anchors, activating only when characters are referred to by a pronoun and conditioning on the preceding frame to propagate visual identity. This selective design avoids unconditional cross-frame conditioning and introduces only 149K additional parameters, a fraction of the cost of memory-bank and LLM-augmented approaches. To further stabilize identity, we incorporate SemDrift (Guided Semantic Drift Correction) applied only during training. When text is vague or referential, the denoiser lacks a visual anchor for identity-defining attributes, causing character appearance to drift across frames, SemDrift corrects this by aligning denoiser representations with pretrained DINOv3 visual embeddings, enforcing semantic identity stability at zero inference cost. ReCap outperforms previous state-of-the-art, StoryGPT-V, on the two main benchmarks for story visualization by 2.63% Character-Accuracy on FlintstonesSV and by 5.65% on PororoSV, establishing a new state-of-the-art character consistency on both benchmarks. Furthermore, we extend story visualization to human-centric narratives derived from real films, demonstrating the capability of ReCap beyond stylized cartoon domains.

Problem

• Story Visualization
• Referential Grounding
• Cross-frame Consistency
• Character Identity
• Visual Coherence

Innovation

• Referential Grounding
• Lightweight Consistency
• CORE Module
• Semantic Drift Correction
• Story Visualization