Open-World Video Segmentation

2026-06-14

AI Summary

Existing video segmentation methods struggle to achieve persistent object discovery and identity consistency in long-duration, ego-motion-rich open-world videos, and conventional 1:1 evaluation protocols fail to fairly assess predictions that are semantically correct but differ in granularity. To address these challenges, this work proposes Savvy—a training-free system that leverages hierarchical mask discovery, delayed admission, and trajectory integration to enable stable object discovery and identity tracking. We introduce the first practical framework for zero-shot long-term open-world video segmentation and propose a granularity-agnostic Open-World Grouping Assessment (OGA) suite, featuring novel metrics such as Identity Persistence (IP) and Identity Concentration (IC). Evaluated on real-world long-duration benchmarks like ScanNet and HM3D, Savvy significantly outperforms strong baselines across STQ, VPQ∞, IP, and IC, demonstrating both methodological efficacy and evaluation robustness.

Abstract

While video segmentation has advanced rapidly on short clips and closed-set benchmarks, open-world video segmentation remains largely unexplored. The challenge is twofold: (1) existing methods are not designed to support object discovery and identity maintenance in long videos of dynamic ego-motion, and (2) existing evaluation protocols rely on a rigid 1:1 matching that unfairly penalizes semantically valid predictions with mismatched granularity. To address both gaps, we introduce Savvy, a practical and strong system for zero-shot open-world long-horizon video segmentation. Savvy combines hierarchical mask discovery, deferred admission, and track consolidation to support persistent object discovery, safe track promotion, and stable long-range identity maintenance. We further propose OGA, a granularity-aware evaluation suite for open-world video segmentation. Built on a Granularity-Agnostic (GA) matching protocol, OGA relaxes conventional 1:1 matching to an n:1 mapping, but still enforces temporal rigor by detecting support discontinuities through sever points and scoring each reference object through its dominant coherent fragment. This prevents fragmented or flickering support from being over-rewarded while enabling GA-adapted metrics and structural diagnostics: identity persistence (IP), and identity concentration (IC). On VIPSeg, we show that standard 1:1 evaluation substantially underestimates open-world methods, whereas GA evaluation recovers much of their suppressed performance. On the more realistic long-horizon benchmarks: ScanNet and HM3D, Savvy consistently outperforms strong baselines across both classical and proposed metrics, including STQ, VPQ$_\infty$, IP and IC. Together, these results establish a practical benchmark and a strong baseline for open-world long-horizon video segmentation.

Problem

• open-world video segmentation
• object discovery
• identity maintenance
• evaluation protocol
• long-horizon video

Innovation

• open-world video segmentation
• long-horizon tracking
• granularity-agnostic evaluation
• identity persistence
• zero-shot segmentation