Generating Visual Stories with Grounded and Coreferent Characters

2024-09-20

arXiv.org

AI Summary

Existing visual story generation methods neglect character modeling, leading to missing character references, ambiguous pronouns, and cross-frame inconsistency. To address this, we introduce the **character-centric visual story generation task**, aiming for visual localizability and textual coreference consistency of characters across multi-image sequences. We construct the first VIST extension dataset annotated with vision–language character coreference chains; design dual-dimensional evaluation metrics—character richness and coreference quality; and propose a multimodal generation framework integrating automated coreference annotation, character-aware fine-tuning, and joint visual grounding with textual coreference modeling. Experiments demonstrate that our approach significantly outperforms baselines and state-of-the-art models in character recurrence rate, visual grounding fidelity, and cross-sentence coreference consistency.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Characters are important in narratives. They move the plot forward, create emotional connections, and embody the story's themes. Visual storytelling methods focus more on the plot and events relating to it, without building the narrative around specific characters. As a result, the generated stories feel generic, with character mentions being absent, vague, or incorrect. To mitigate these issues, we introduce the new task of character-centric story generation and present the first model capable of predicting visual stories with consistently grounded and coreferent character mentions. Our model is finetuned on a new dataset which we build on top of the widely used VIST benchmark. Specifically, we develop an automated pipeline to enrich VIST with visual and textual character coreference chains. We also propose new evaluation metrics to measure the richness of characters and coreference in stories. Experimental results show that our model generates stories with recurring characters which are consistent and coreferent to larger extent compared to baselines and state-of-the-art systems.

Problem

• Generates visual stories with consistent character mentions.
• Addresses generic storytelling by focusing on character-centric narratives.
• Introduces new metrics to evaluate character richness and coreference.

Innovation

• Character-centric story generation model
• Automated pipeline for coreference chains
• New metrics for character richness evaluation