SAM2Matting: Generalized Image and Video Matting

2026-06-25

AI Summary

Existing video matting methods struggle to simultaneously achieve temporal consistency and cross-domain generalization due to their reliance on costly, specialized datasets, and often fail to jointly handle high-level semantic tracking and low-level fine details. To address these limitations, this work proposes SAM2Matting—a decoupled tracker-to-matting framework that leverages foundation trackers like SAM2 to ensure temporal coherence, employs a region proposal module to bridge tracking and matting, and introduces a dedicated matting head to recover fine-grained details. Remarkably, SAM2Matting achieves state-of-the-art performance in video matting using only static image training data, supports diverse prompt inputs, and demonstrates superior generalization and temporal stability across both portrait and complex real-world scenes.

Abstract

Despite impressive advances in image matting, video matting remains challenging due to the inherent gap between high-level tracking, which requires frame-wise understanding, and low-level matting, which focuses on extremely fine-grained details. Existing methods attempt this with expensive and narrowly-scoped video matting datasets, which may limit out-of-domain generalization and compromise tracking robustness. We rethink the paradigm with SAM2Matting, a tracker-to-matting framework that advances VOS trackers to high-fidelity video matting. Specifically, it decouples the task by enhancing a foundational tracker (e.g., SAM2, SAM3) with a region-proposal bridge and dedicated matting heads, enabling the uncompromised tracker to handle temporal consistency while the matting components resolve fine-grained details. Notably, despite being trained only on images, SAM2Matting establishes new state-of-the-art performance on video matting, supports diverse prompt types, maintains strong temporal consistency, and demonstrates robust generalization across both human-centric and in-the-wild scenarios.

Problem

• video matting
• temporal consistency
• out-of-domain generalization
• fine-grained details
• tracking robustness

Innovation

• video matting
• tracker-to-matting
• temporal consistency
• foundation model adaptation
• generalization