FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spaces

2025-01-22

AI Summary

To address the complexity of cinematic production in 3D virtual worlds, high manual decision-making costs, and severe hallucination issues in large language models (LLMs), this paper proposes the first multi-agent LLM framework tailored for film and television creation. Methodologically, it instantiates specialized agents—including director, screenwriter, actor, and cinematographer—integrated via prompt engineering, iterative refinement protocols, and 3D-scene instruction alignment to enable end-to-end generation from textual concepts to render-ready camera instructions. Its key contributions lie in role-based functional decomposition and a closed-loop feedback mechanism, which jointly mitigate hallucination and significantly enhance narrative coherence and controllability. Human evaluation across 15 creative themes yields a score of 3.98/5, outperforming single-agent baselines (including o1) and Sora. Results validate the efficacy and superiority of multi-agent collaboration in tackling complex creative tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Modeling other Agents
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Virtual film production requires intricate decision-making processes, including scriptwriting, virtual cinematography, and precise actor positioning and actions. Motivated by recent advances in automated decision-making with language agent-based societies, this paper introduces FilmAgent, a novel LLM-based multi-agent collaborative framework for end-to-end film automation in our constructed 3D virtual spaces. FilmAgent simulates various crew roles, including directors, screenwriters, actors, and cinematographers, and covers key stages of a film production workflow: (1) idea development transforms brainstormed ideas into structured story outlines; (2) scriptwriting elaborates on dialogue and character actions for each scene; (3) cinematography determines the camera setups for each shot. A team of agents collaborates through iterative feedback and revisions, thereby verifying intermediate scripts and reducing hallucinations. We evaluate the generated videos on 15 ideas and 4 key aspects. Human evaluation shows that FilmAgent outperforms all baselines across all aspects and scores 3.98 out of 5 on average, showing the feasibility of multi-agent collaboration in filmmaking. Further analysis reveals that FilmAgent, despite using the less advanced GPT-4o model, surpasses the single-agent o1, showing the advantage of a well-coordinated multi-agent system. Lastly, we discuss the complementary strengths and weaknesses of OpenAI's text-to-video model Sora and our FilmAgent in filmmaking.

Problem

• 3D virtual worlds
• automated filmmaking
• decision-making reduction

Innovation

• Automated Filmmaking
• Multi-Role Collaboration
• Large Language Model