UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating

2026-06-19

AI Summary

This work addresses the challenge of maintaining cross-shot consistency in multi-shot audiovisual generation, where existing methods struggle to preserve subject appearance, scene context, and speaker identity while lacking scalability and effective memory mechanisms. Building upon the LTX-2.3 architecture, the authors propose a boundary-aware gating mechanism and fixed-size long- and short-term memory slots (LTM/STM) that integrate visual cut probabilities with beat signals during shot transitions. Speaker identity consistency is achieved by injecting reference speaker tokens, eliminating the need for a sliding audio bank, and a learnable discrete prior over shot types modulates transition intensity. The method significantly outperforms open-source baselines under I2V, T2V, and R2V settings, achieving cross-shot coherence on par with the strongest closed-source systems, and introduces a multicultural multi-shot benchmark dataset comprising 200 sequences across six ethnic groups and over ten languages.

Abstract

Generating a coherent multi-shot video requires structured cross-shot memory. Subject appearance, scene context, and speaker identity must persist across cuts. Existing approaches either train end-to-end over fixed-length sequences and cannot scale, generate shot-by-shot with memory banks that grow linearly, or orchestrate pretrained generators under an LLM planner without a multi-shot-aware backbone. We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3, trained on annotated cinematic and music-video shots. The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail, both updated at every cut by a boundary-conditioned gate that fuses visual cut probability and beat-tracker signals. The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank. A discrete cut-type prior, learned through AdaLN, becomes an inference-time control knob over transition strength. We release a benchmark of $200$ multi-cultural multi-shot sequences spanning six ethnic regions and ten or more languages, with per-shot reference identities, reference audio, and per-boundary transition labels. Evaluated across I2V, T2V, and R2V conditioning modes, UnityShots leads open-source baselines on every cross-shot coherence metric and matches the strongest closed-source system on the multi-shot axes.

Problem

• multi-shot generation
• cross-shot coherence
• audio-video generation
• memory persistence
• shot boundary

Innovation

• memory-driven generation
• boundary-aware gating
• multi-shot coherence
• fixed-size memory slots
• reference speaker token