Facilitating Video Story Interaction with Multi-Agent Collaborative System

2025-05-02

AI Summary

Existing video storytelling interaction methods rely on predefined narratives, lacking personalization and dynamic evolution capabilities. This paper proposes a multi-agent collaborative interactive system for video narrative generation, featuring a three-tier architecture integrating vision-language models (VLMs), retrieval-augmented generation (RAG), and multi-agent systems (MAS). The system enables user-intent-driven character development, emergent social behaviors, and on-demand scene expansion. Departing from passive viewing paradigms, it supports cross-modal narrative understanding and generation, endowing characters with long-term memory and explicit social relationship modeling. Evaluated on the *Harry Potter* series, generated characters exhibit significant developmental progression and social coherence, while narrative scenes support high-fidelity, customizable visual interaction. To our knowledge, this work is the first to deeply couple RAG with MAS within a video narrative framework, establishing a novel paradigm for personalized, dynamically evolving interactive video storytelling.

Technology Category

• Multiagent Systems: Multiagent Learning
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Video story interaction enables viewers to engage with and explore narrative content for personalized experiences. However, existing methods are limited to user selection, specially designed narratives, and lack customization. To address this, we propose an interactive system based on user intent. Our system uses a Vision Language Model (VLM) to enable machines to understand video stories, combining Retrieval-Augmented Generation (RAG) and a Multi-Agent System (MAS) to create evolving characters and scene experiences. It includes three stages: 1) Video story processing, utilizing VLM and prior knowledge to simulate human understanding of stories across three modalities. 2) Multi-space chat, creating growth-oriented characters through MAS interactions based on user queries and story stages. 3) Scene customization, expanding and visualizing various story scenes mentioned in dialogue. Applied to the Harry Potter series, our study shows the system effectively portrays emergent character social behavior and growth, enhancing the interactive experience in the video story world.

Problem

• Enhancing video story interaction through multi-agent collaboration
• Overcoming limitations in user selection and narrative customization
• Enabling dynamic character growth and scene visualization

Innovation

• Uses Vision Language Model for video story understanding
• Combines RAG and Multi-Agent System for dynamic interactions
• Customizes scenes and characters based on user queries