OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video Diffusion Models

2025-03-23

AI Summary

This work addresses the problem of unsupervised, real-time hierarchical decomposition of input videos into semantically consistent layers—namely background, foreground objects, and physically grounded effects (e.g., shadows, reflections)—without any fine-tuning or training. The proposed method leverages a pre-trained video diffusion model and introduces a novel zero-shot adaptation mechanism to achieve Omnimatte. It precisely localizes and reconstructs object footprints via self-attention maps and directly disentangles and recombines semantic layers through latent-space arithmetic operations. The approach enables practical applications including object removal, extraction, and seamless cross-video recomposition. Experiments demonstrate superior background reconstruction quality over state-of-the-art methods while achieving real-time inference speeds (as low as <35 ms per frame), establishing it as the fastest zero-shot Omnimatte solution to date.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Omnimatte aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often require extensive training or costly self-supervised optimization. In this paper, we present OmnimatteZero, a training-free approach that leverages off-the-shelf pre-trained video diffusion models for omnimatte. It can remove objects from videos, extract individual object layers along with their effects, and composite those objects onto new videos. We accomplish this by adapting zero-shot image inpainting techniques for video object removal, a task they fail to handle effectively out-of-the-box. We then show that self-attention maps capture information about the object and its footprints and use them to inpaint the object's effects, leaving a clean background. Additionally, through simple latent arithmetic, object layers can be isolated and recombined seamlessly with new video layers to produce new videos. Evaluations show that OmnimatteZero not only achieves superior performance in terms of background reconstruction but also sets a new record for the fastest Omnimatte approach, achieving real-time performance with minimal frame runtime.

Problem

• Decompose video into layers with effects without training
• Remove objects and composite them onto new videos
• Achieve real-time performance with minimal frame runtime

Innovation

• Training-free approach using pre-trained video models
• Zero-shot inpainting for video object removal
• Latent arithmetic for seamless object recombination