Online Segment 3D Gaussians via Launching Virtual Drones

2026-07-01

AI Summary

Existing interactive segmentation methods for 3D Gaussian scenes rely on preprocessing steps that take tens of seconds to minutes, hindering their applicability in online settings. This work proposes SAGO, a framework that achieves real-time, preprocessing-free 3D Gaussian segmentation for the first time. By modeling the segmentation task as an online optimal viewpoint planning problem framed within a Markov decision process—enabled by a virtual drone—SAGO attains sub-second latency (<1 second) without any prior preprocessing. The method accelerates segmentation by over 50× compared to existing preprocessing-free approaches and efficiently extracts clean 3D assets, significantly advancing downstream applications such as real-time object manipulation and scene editing.

Abstract

Interactive segmentation of 3D Gaussians offers a compelling opportunity for real-time manipulation of 3D scenes, thanks to the real-time rendering capability of 3D Gaussian Splatting (3DGS). However, existing methods require a time-consuming per-scene setup - typically tens of seconds or even minutes - before interactive segmentation can begin on a raw 3DGS scene. This setup involves multi-view mask preparation, mask lifting, and feature distillation, creating a major bottleneck for online applications. To address this limitation, we aim to completely eliminate the setup stage for interactive 3DGS segmentation while keeping the segmentation time practical (under 1 second). In this work, we present SAGO (Segment Any Gaussians Online), a novel setup-free framework for interactive 3DGS segmentation. By introducing virtual drones, our method reframes the 3D segmentation problem as an online Next-Best-View (NBV) planning task formulated within a Markov process. Extensive experiments demonstrate that SAGO can extract clean 3D assets directly from 3D Gaussians with sub-second latency, thereby enabling a broad range of downstream applications such as object manipulation and scene editing. Moreover, our method achieves over a 50x speedup compared to the previous setup-free 3DGS segmentation frameworks.

Problem

• 3D Gaussian Splatting
• interactive segmentation
• online segmentation
• setup-free
• Next-Best-View

Innovation

• 3D Gaussian Splatting
• interactive segmentation
• virtual drones
• Next-Best-View planning
• setup-free