Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation

2026-04-19

AI Summary

Existing video-to-music generation methods rely on a single visual condition, limiting fine-grained control over both semantic content and stylistic attributes. This work proposes a novel framework that integrates autoregressive planning with diffusion-based synthesis: a text-guided autoregressive module first generates high-level musical latent variables to enable intention-driven semantic control, followed by a local diffusion transformer that synthesizes high-fidelity audio. By introducing intention-driven autoregressive planning into video-to-music generation for the first time, the approach effectively co-optimizes semantic alignment with audio realism. Experimental results demonstrate that the model outperforms current state-of-the-art methods on both in-distribution and out-of-distribution benchmarks, while achieving a 2.21× faster inference speed.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Video-to-music (V2M) is the fundamental task of creating background music for an input video. Recent V2M models achieve audiovisual alignment by typically relying on visual conditioning alone and provide limited semantic and stylistic controllability to the end user. In this paper, we present Video-Robin, a novel text-conditioned video-to-music generation model that enables fast, high-quality, semantically aligned music generation for video content. To balance musical fidelity and semantic understanding, Video-Robin integrates autoregressive planning with diffusion-based synthesis. Specifically, an autoregressive module models global structure by semantically aligning visual and textual inputs to produce high-level music latents. These latents are subsequently refined into coherent, high-fidelity music using local Diffusion Transformers. By factoring semantically driven planning into diffusion-based synthesis, Video-Robin enables fine-grained creator control without sacrificing audio realism. Our proposed model outperforms baselines that solely accept video input and additional feature conditioned baselines on both in-distribution and out-of-distribution benchmarks with a 2.21x speed in inference compared to SOTA. We will open-source everything upon paper acceptance.

Problem

• video-to-music
• semantic alignment
• controllability
• audiovisual generation
• music generation

Innovation

• autoregressive planning
• diffusion-based synthesis
• text-conditioned video-to-music
• semantic alignment
• Diffusion Transformers