Tora3: Trajectory-Guided Audio-Video Generation with Physical Coherence

2026-04-10

AI Summary

Existing audio-visual generation methods struggle to model coherent relationships between motion and sound, often resulting in unstable dynamics and loose audio-visual alignment. To address this, this work proposes Tora3, a novel framework that explicitly leverages object trajectories as a shared kinematic prior across modalities. By integrating trajectory-guided motion representations, an acoustic alignment module grounded in second-order trajectory states, and a hybrid flow-matching mechanism, Tora3 enables joint generation of visual motion and audio events with improved coherence. The model is trained on PAV, a newly curated large-scale dataset with trajectory annotations, and demonstrates significant improvements over existing open-source baselines in motion realism, audio-visual synchronization, and overall generation quality.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Multimodal Learning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Audio-video (AV) generation has recently made strong progress in perceptual quality and multimodal coherence, yet generating content with plausible motion-sound relations remains challenging. Existing methods often produce object motions that are visually unstable and sounds that are only loosely aligned with salient motion or contact events, largely because they lack an explicit motion-aware structure shared by video and audio generation. We present Tora3, a trajectory-guided AV generation framework that improves physical coherence by using object trajectories as a shared kinematic prior. Rather than treating trajectories as a video-only control signal, Tora3 uses them to jointly guide visual motion and acoustic events. Specifically, we design a trajectory-aligned motion representation for video, a kinematic-audio alignment module driven by trajectory-derived second-order kinematic states, and a hybrid flow matching scheme that preserves trajectory fidelity in trajectory-conditioned regions while maintaining local coherence elsewhere. We further curate PAV, a large-scale AV dataset emphasizing motion-relevant patterns with automatically extracted motion annotations. Extensive experiments show that Tora3 improves motion realism, motion-sound synchronization, and overall AV generation quality over strong open-source baselines.

Problem

• audio-video generation
• motion-sound alignment
• physical coherence
• object trajectories
• multimodal coherence

Innovation

• trajectory-guided generation
• physical coherence
• kinematic-audio alignment
• multimodal AV generation
• hybrid flow matching