SAM3-LiteText: An Anatomical Study of the SAM3 Text Encoder for Efficient Vision-Language Segmentation

2026-02-12

AI Summary

This work addresses the significant redundancy in generic text encoders of current vision–language segmentation models, which are ill-suited for the practical demands of short, structured prompts. Through a systematic analysis of over 400,000 real-world prompts, the study reveals— for the first time—that their textual representations exhibit a low-dimensional manifold structure and high sparsity. Leveraging these insights, the authors propose a task-customized, minimalist text encoding architecture. By replacing the original large-scale encoder with a MobileCLIP-based student model trained via knowledge distillation, the approach achieves comparable performance on both image and video segmentation tasks while reducing the text encoder’s parameter count by 88%, substantially lowering static memory consumption.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-language segmentation models such as SAM3 enable flexible, prompt-driven visual grounding, but inherit large, general-purpose text encoders originally designed for open-ended language understanding. In practice, segmentation prompts are short, structured, and semantically constrained, leading to substantial over-provisioning in text encoder capacity and persistent computational and memory overhead. In this paper, we perform a large-scale anatomical analysis of text prompting in vision-language segmentation, covering 404,796 real prompts across multiple benchmarks. Our analysis reveals severe redundancy: most context windows are underutilized, vocabulary usage is highly sparse, and text embeddings lie on low-dimensional manifold despite high-dimensional representations. Motivated by these findings, we propose SAM3-LiteText, a lightweight text encoding framework that replaces the original SAM3 text encoder with a compact MobileCLIP student that is optimized by knowledge distillation. Extensive experiments on image and video segmentation benchmarks show that SAM3-LiteText reduces text encoder parameters by up to 88%, substantially reducing static memory footprint, while maintaining segmentation performance comparable to the original model. Code: https://github.com/SimonZeng7108/efficientsam3/tree/sam3_litetext.

Problem

• vision-language segmentation
• text encoder redundancy
• prompt efficiency
• computational overhead
• model compression

Innovation

• vision-language segmentation
• text encoder compression
• knowledge distillation
• prompt redundancy
• MobileCLIP