DiffusionCinema: Text-to-Aerial Cinematography

2026-01-24

AI Summary

This work addresses the challenge of efficiently translating natural language descriptions of cinematic intent into precise and smooth UAV flight trajectories, a task traditionally hindered by reliance on manual piloting. The paper introduces the first end-to-end text-to-aerial-shot generation framework, which leverages a diffusion model to jointly process natural language prompts and an initial visual snapshot, automatically synthesizing spatiotemporal flight paths that respect both scene geometry and cinematographic semantics to autonomously guide drone filming. By treating the diffusion model as a “creative operator,” this approach establishes a novel paradigm for text-driven aerial cinematography. User studies demonstrate that, compared to conventional remote control, the system significantly reduces overall workload (21.6 vs. 58.1), mental demand (11.5 vs. 60.5), and frustration (14.0 vs. 54.5), confirming its usability and effectiveness.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We propose a novel Unmanned Aerial Vehicles (UAV) assisted creative capture system that leverages diffusion models to interpret high-level natural language prompts and automatically generate optimal flight trajectories for cinematic video recording. Instead of manually piloting the drone, the user simply describes the desired shot (e.g.,"orbit around me slowly from the right and reveal the background waterfall"). Our system encodes the prompt along with an initial visual snapshot from the onboard camera, and a diffusion model samples plausible spatio-temporal motion plans that satisfy both the scene geometry and shot semantics. The generated flight trajectory is then executed autonomously by the UAV to record smooth, repeatable video clips that match the prompt. User evaluation using NASA-TLX showed a significantly lower overall workload with our interface (M = 21.6) compared to a traditional remote controller (M = 58.1), demonstrating a substantial reduction in perceived effort. Mental demand (M = 11.5 vs. 60.5) and frustration (M = 14.0 vs. 54.5) were also markedly lower for our system, confirming clear usability advantages in autonomous text-driven flight control. This project demonstrates a new interaction paradigm: text-to-cinema flight, where diffusion models act as the"creative operator"converting story intentions directly into aerial motion.

Problem

• text-to-aerial cinematography
• UAV
• diffusion models
• flight trajectory generation
• natural language interface

Innovation

• diffusion models
• text-to-cinematography
• autonomous UAV
• natural language interface
• aerial motion planning