Kaleido: Open-Sourced Multi-Subject Reference Video Generation Model

2025-10-21

AI Summary

Existing subject-to-video generation methods struggle to simultaneously ensure subject consistency and background disentanglement under multiple image references, resulting in low reference fidelity and severe semantic drift. To address this, we propose the first end-to-end framework for multi-subject-consistent video generation. Our approach comprises three key components: (1) constructing a high-quality, cross-reference-free multi-reference training dataset; (2) introducing Reference-aware Rotational Positional Encoding (R-RoPE) to enable stable alignment and precise fusion of multi-image features; and (3) incorporating low-quality sample filtering, diverse synthetic data augmentation, and cross-instance disentanglement to enhance generalization. Extensive evaluations across multiple benchmarks demonstrate that our method significantly outperforms state-of-the-art approaches in subject consistency, reference fidelity, and semantic stability—particularly excelling in complex multi-subject scenarios with strong robustness and generalization capability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Consent frameworks and practices on the web

Abstract

We present Kaleido, a subject-to-video~(S2V) generation framework, which aims to synthesize subject-consistent videos conditioned on multiple reference images of target subjects. Despite recent progress in S2V generation models, existing approaches remain inadequate at maintaining multi-subject consistency and at handling background disentanglement, often resulting in lower reference fidelity and semantic drift under multi-image conditioning. These shortcomings can be attributed to several factors. Primarily, the training dataset suffers from a lack of diversity and high-quality samples, as well as cross-paired data, i.e., paired samples whose components originate from different instances. In addition, the current mechanism for integrating multiple reference images is suboptimal, potentially resulting in the confusion of multiple subjects. To overcome these limitations, we propose a dedicated data construction pipeline, incorporating low-quality sample filtering and diverse data synthesis, to produce consistency-preserving training data. Moreover, we introduce Reference Rotary Positional Encoding (R-RoPE) to process reference images, enabling stable and precise multi-image integration. Extensive experiments across numerous benchmarks demonstrate that Kaleido significantly outperforms previous methods in consistency, fidelity, and generalization, marking an advance in S2V generation.

Problem

• Maintaining multi-subject consistency in video generation
• Handling background disentanglement to prevent semantic drift
• Improving reference fidelity under multi-image conditioning

Innovation

• Data pipeline filters low-quality samples and synthesizes diverse data
• Reference Rotary Positional Encoding enables stable multi-image integration
• Framework maintains multi-subject consistency and background disentanglement