VideoMaMa: Mask-Guided Video Matting via Generative Prior

2026-01-20

AI Summary

This work addresses the challenges of poor generalization and scarce annotated data in video matting under real-world conditions by proposing a zero-shot video matting method that, for the first time, integrates generative priors from pretrained video diffusion models with coarse segmentation cues to produce high-fidelity alpha mattes. To facilitate research and evaluation, the authors introduce a scalable pseudo-labeling pipeline and release MA-V, a large-scale real-world video matting dataset comprising over 50,000 high-quality annotated videos. Leveraging this dataset, they fine-tune SAM2 to obtain SAM2-Matte, which significantly outperforms existing approaches on in-the-wild video sequences.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance

Abstract

Generalizing video matting models to real-world videos remains a significant challenge due to the scarcity of labeled data. To address this, we present Video Mask-to-Matte Model (VideoMaMa) that converts coarse segmentation masks into pixel accurate alpha mattes, by leveraging pretrained video diffusion models. VideoMaMa demonstrates strong zero-shot generalization to real-world footage, even though it is trained solely on synthetic data. Building on this capability, we develop a scalable pseudo-labeling pipeline for large-scale video matting and construct the Matting Anything in Video (MA-V) dataset, which offers high-quality matting annotations for more than 50K real-world videos spanning diverse scenes and motions. To validate the effectiveness of this dataset, we fine-tune the SAM2 model on MA-V to obtain SAM2-Matte, which outperforms the same model trained on existing matting datasets in terms of robustness on in-the-wild videos. These findings emphasize the importance of large-scale pseudo-labeled video matting and showcase how generative priors and accessible segmentation cues can drive scalable progress in video matting research.

Problem

• video matting
• generalization
• labeled data scarcity
• real-world videos
• alpha matte

Innovation

• video matting
• generative prior
• zero-shot generalization
• pseudo-labeling
• diffusion model