Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation

2026-04-28

AI Summary

This work addresses the challenges of joint modeling and inefficient autoregressive inference in long-duration audio-visual synchronization generation by proposing a two-stage training strategy combined with a Mutual Forcing framework. The approach integrates few-step and multi-step generation modes within a single weight-shared model, leveraging a self-distillation mechanism to ensure consistency between training and inference. It directly utilizes real paired data for self-evolution without requiring an additional bidirectional teacher model. The method supports flexible training sequence lengths and achieves comparable or superior generation quality to strong baselines that require approximately 50 sampling steps, while needing only 4–8 steps—demonstrating significant improvements in both generation fidelity and inference efficiency.

Abstract

In this work, we propose Mutual Forcing, a framework for fast autoregressive audio-video generation with long-horizon audio-video synchronization. Our approach addresses two key challenges: joint audio-video modeling and fast autoregressive generation. To ease joint audio-video optimization, we adopt a two-stage training strategy: we first train uni-modal generators and then couple them into a unified audio-video model for joint training on paired data. For streaming generation, we ask whether a native fast causal audio-video model can be trained directly, instead of following existing streaming distillation pipelines that typically train a bidirectional model first and then convert it into a causal generator through multiple distillation stages. Our answer is Mutual Forcing, which builds directly on native autoregressive model and integrates few-step and multi-step generation within a single weight-shared model, enabling self-distillation and improved training-inference consistency. The multi-step mode improves the few-step mode via self-distillation, while the few-step mode generates historical context during training to improve training-inference consistency; because the two modes share parameters, these two effects reinforce each other within a single model. Compared with prior approaches such as Self-Forcing, Mutual Forcing removes the need for an additional bidirectional teacher model, supports more flexible training sequence lengths, reduces training overhead, and allows the model to improve directly from real paired data rather than a fixed teacher. Experiments show that Mutual Forcing matches or surpasses strong baselines that require around 50 sampling steps while using only 4 to 8 steps, demonstrating substantial advantages in both efficiency and quality. The project page is available at https://mutualforcing.github.io.

Problem

• audio-video synchronization
• autoregressive generation
• joint audio-video modeling
• fast generation
• character generation

Innovation

• Mutual Forcing
• autoregressive generation
• audio-video synchronization
• self-distillation
• fast inference