PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animation

2025-06-01

AI Summary

In VFX production, 3D effects traditionally rely on specialized software, entail lengthy manual workflows, and existing generative approaches—such as diffusion models—suffer from high computational overhead and inefficient 4D spatiotemporal inference. Method: This paper proposes a text-driven, real-time 3D Gaussian animation framework. It formulates VFX creation as a time-varying 4D flow-field prediction task, integrates LLMs and VLMs for functional text understanding, and employs differentiable 3D Gaussian splatting to construct a lightweight, parameterized flow-field model that maps natural-language instructions (e.g., “vase emits orange light then explodes”) end-to-end to dynamic color, opacity, and spatial trajectories. Contribution/Results: We present the first browser-level, real-time language-driven volumetric animation system; it eliminates the need for mesh extraction, physics simulation, or manual rigging. Deployable on consumer-grade hardware and web browsers, it synthesizes high-fidelity dynamic VFX from single-sentence prompts—significantly lowering the barrier to professional-grade VFX authoring and reducing labor costs.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Visual effects (VFX) are key to immersion in modern films, games, and AR/VR. Creating 3D effects requires specialized expertise and training in 3D animation software and can be time consuming. Generative solutions typically rely on computationally intense methods such as diffusion models which can be slow at 4D inference. We reformulate 3D animation as a field prediction task and introduce a text-driven framework that infers a time-varying 4D flow field acting on 3D Gaussians. By leveraging large language models (LLMs) and vision-language models (VLMs) for function generation, our approach interprets arbitrary prompts (e.g.,"make the vase glow orange, then explode") and instantly updates color, opacity, and positions of 3D Gaussians in real time. This design avoids overheads such as mesh extraction, manual or physics-based simulations and allows both novice and expert users to animate volumetric scenes with minimal effort on a consumer device even in a web browser. Experimental results show that simple textual instructions suffice to generate compelling time-varying VFX, reducing the manual effort typically required for rigging or advanced modeling. We thus present a fast and accessible pathway to language-driven 3D content creation that can pave the way to democratize VFX further.

Problem

• Creating 3D VFX requires expertise and is time-consuming
• Existing generative methods are slow for 4D inference
• Manual effort for rigging and advanced modeling is high

Innovation

• Text-driven 4D flow field prediction
• LLMs and VLMs for function generation
• Real-time 3D Gaussian updates