Autoregressive Modeling of Film with Applications in Video Montage

2026-07-16

AI Summary

This work addresses the problem of video montage—automatically editing raw footage into coherent cinematic sequences. It proposes FilmGPT, the first framework to adapt the autoregressive modeling paradigm of large language models to video editing. Built upon a long-context Transformer architecture, FilmGPT is pretrained on a large-scale film corpus to learn the “grammatical” structure of shot sequences. The authors introduce a source-constrained decoding algorithm that selects an optimal sequence from given shots without generating new frames. Requiring no handcrafted rules, the method significantly outperforms existing approaches, achieving state-of-the-art performance on standard ordering benchmarks and in user studies. It has been successfully deployed in applications such as automatic clip trimming and human-in-the-loop collaborative editing.

Abstract

This work introduces FilmGPT, an autoregressive transformer designed to address the challenge of video montage--turning a collection of raw, "unwatchable" footage into coherent cinematic sequences. Inspired by language learning in modern LLMs, we train a long-context autoregressive transformer on a large corpus of movies. The aim is to implicitly capture the "grammar" of film directly from data rather than from hand-coded rules. Unlike other generative models, FilmGPT does not generate any new video frames. Instead, at inference time, we introduce a footage-constrained decoding algorithm to select the best next shot from the input raw footage according to the statistical patterns learned from films. We first evaluate these learned statistics directly by using the FilmGPT autoregressive model for next shot prediction on a standard benchmark of shot sequence ordering, outperforming the previous state of the art. We then evaluate our footage-constrained decoding algorithm on the full film editing task via a user study, and find that our FilmGPT-based editing significantly outperforms previous approaches. Finally, we demonstrate the applicability of FilmGPT to a wide range of applications in video montage, from automatic video segment trimming to human-in-the-loop film editing.

Problem

• video montage
• film editing
• autoregressive modeling
• shot selection
• cinematic coherence

Innovation

• autoregressive modeling
• video montage
• film grammar
• footage-constrained decoding
• transformer