ActiveSAM: Image-Conditional Class Pruning for Fast and Accurate Open-Vocabulary Segmentation

2026-06-15

AI Summary

This work addresses the inefficiency of directly applying Segment Anything Model 3 (SAM3) to open-vocabulary semantic segmentation, where high-resolution decoding over the entire vocabulary is wasteful since each image contains only a few active categories. To overcome this, the authors propose ActiveSAM, a training-free, zero-shot inference framework that achieves image-adaptive vocabulary pruning for the first time. By leveraging low-resolution category presence previews, prompt normalization and expansion, bucketed feature reuse, and margin-aware background calibration, ActiveSAM performs high-resolution decoding exclusively on potentially active categories while keeping the SAM3 backbone frozen. Evaluated across eight benchmarks, ActiveSAM improves average mIoU by 1.4%, accelerates inference by up to 5.5×, and significantly enhances robustness to image perturbations.

Abstract

Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted segmentation, but applying it directly to open-vocabulary semantic segmentation (OVSS) is inefficient: full-resolution decoding is typically run over the entire dataset vocabulary, whereas each image contains only a small active subset of classes. We introduce ActiveSAM, a training-free, zero-shot inference framework that turns SAM 3 into an active-vocabulary segmenter. ActiveSAM first canonicalizes and expands class prompts, then estimates an image-conditioned active set from a low-resolution presence preview. Only the retained classes are decoded at full resolution, using bucketed prompt multiplexing with the frozen SAM 3 decoder. The preview stage uses only class-presence evidence and skips unnecessary segmentation-head computation, while the final stage applies margin-aware background calibration to suppress low-confidence pixels. ActiveSAM requires no target-dataset training, no weight updates, and no oracle class-presence labels. Across eight OVSS benchmarks, ActiveSAM improves the speed-accuracy tradeoff of training-free open-vocabulary semantic segmentation, outperforming the current state-of-the-art SegEarth-OV3 by approximately +1.4 mIoU on average while running up to 5.5x faster on large-vocabulary datasets. ActiveSAM also demonstrates the strongest robustness under image corruption that simulates real-world distribution shift, making it well-suited for deployment in noisy-input domains such as autonomous driving and embodied AI. Code is available at https://github.com/VILA-Lab/ActiveSAM.

Problem

• open-vocabulary semantic segmentation
• class pruning
• efficient inference
• image-conditioned active vocabulary
• zero-shot segmentation

Innovation

• open-vocabulary segmentation
• class pruning
• zero-shot inference
• SAM
• efficient decoding