LensCraft: Your Professional Virtual Cinematographer

2025-06-01

AI Summary

Digital creators face a longstanding challenge in translating abstract creative concepts into cinematic camera motions with precision: existing automated systems typically model subjects as point masses, neglecting their 3D orientation and physical volume—leading to spatial perceptual distortions. This paper introduces the first neural cinematography method that jointly incorporates cinematic priors and explicit subject volume–orientation modeling. We propose a conditional trajectory generation framework supporting multimodal inputs—including text, reference trajectories, and keypoint annotations—and design a high-fidelity simulation training environment alongside a real-time, lightweight inference architecture. Our approach achieves significant improvements over state-of-the-art methods on both static and dynamic scenes, setting new benchmarks in motion accuracy and narrative coherence. To foster reproducibility and community advancement, we publicly release a curated dataset, the simulation platform, pretrained model weights, and source code.

Technology Category

• Computer Vision: Motion & Tracking
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Natural Language Processing: Generation

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Web data generation and simulation
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Digital creators, from indie filmmakers to animation studios, face a persistent bottleneck: translating their creative vision into precise camera movements. Despite significant progress in computer vision and artificial intelligence, current automated filming systems struggle with a fundamental trade-off between mechanical execution and creative intent. Crucially, almost all previous works simplify the subject to a single point-ignoring its orientation and true volume-severely limiting spatial awareness during filming. LensCraft solves this problem by mimicking the expertise of a professional cinematographer, using a data-driven approach that combines cinematographic principles with the flexibility to adapt to dynamic scenes in real time. Our solution combines a specialized simulation framework for generating high-fidelity training data with an advanced neural model that is faithful to the script while being aware of the volume and dynamic behavior of the subject. Additionally, our approach allows for flexible control via various input modalities, including text prompts, subject trajectory and volume, key points, or a full camera trajectory, offering creators a versatile tool to guide camera movements in line with their vision. Leveraging a lightweight real time architecture, LensCraft achieves markedly lower computational complexity and faster inference while maintaining high output quality. Extensive evaluation across static and dynamic scenarios reveals unprecedented accuracy and coherence, setting a new benchmark for intelligent camera systems compared to state-of-the-art models. Extended results, the complete dataset, simulation environment, trained model weights, and source code are publicly accessible on LensCraft Webpage.

Problem

• Automated filming systems lack creative intent and mechanical execution balance
• Previous works oversimplify subjects, ignoring orientation and volume during filming
• Existing solutions fail to adapt dynamically to real-time scene changes

Innovation

• Data-driven approach combining cinematographic principles
• Specialized simulation framework for high-fidelity training
• Lightweight real-time architecture for fast inference