AVControl: Efficient Framework for Training Audio-Visual Controls

2026-03-25

AI Summary

Existing audio-visual generation methods struggle to efficiently support diverse control signals, often relying on fixed models or requiring costly architectural modifications. This work proposes a lightweight and extensible framework built upon the LTX-2 joint audio-visual foundation model, where each control modality—such as depth, pose, camera trajectory, sparse motion, and audio—is trained as an independent LoRA module. These modules are integrated into the backbone network via a parallel canvas mechanism that injects them as additional attention tokens, eliminating the need for any architectural changes. This approach enables modular audio-visual control for the first time, allowing flexible composition of control signals and efficient training. On the VACE benchmark, the method outperforms existing approaches in depth- and pose-guided generation, image inpainting, and extrapolation tasks, achieves competitive performance in camera control and audio-visual synthesis, and significantly reduces training costs.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.

Problem

• audio-visual generation
• control modalities
• model architecture
• multimodal control
• efficient training

Innovation

• AVControl
• LoRA
• audio-visual generation
• parallel canvas
• modular control