SparseSAM: Structured Sparsification of Activations in Segment Anything Models

2026-05-17

AI Summary

This work addresses the high latency and substantial memory overhead of the Vision Transformer (ViT) encoder in the Segment Anything Model (SAM) during inference, where existing compression techniques often suffer from significant accuracy degradation or introduce additional computational costs under high compression ratios. The authors propose a training-free, structured activation sparsification framework that jointly accelerates both attention and MLP layers. Specifically, they introduce Stripe-Sort Attention—a hardware-friendly static sparse attention mechanism based on Z-order sorting—and a residual-consistency-aware MLP routing strategy that selectively computes only the most informative tokens, thereby avoiding the overhead of dynamic masking. Evaluated across four segmentation benchmarks, the method achieves high-fidelity acceleration: at a density of 0.4, it incurs only a 0.004 mIoU drop, reducing accuracy loss by 2.10× compared to state-of-the-art token merging approaches, while delivering 2× faster inference and 2.8× memory savings.

Abstract

The Segment Anything Model (SAM) achieves strong open-vocabulary segmentation, but its ViT-based image encoders dominate inference latency and memory. Existing activation compression methods, such as token merging, reduce the token length to process, yet introduce non-trivial runtime overhead and encounter catastrophic quality drop under high compression. Other methods applying Sparse Attention focus on attention alone, leaving the MLP fully dense and capping achievable speedup. We propose SparseSAM, a (i) training-free structured sparsification framework that jointly accelerates attention and MLP layers while preserving token identity. SparseSAM introduces (ii) Stripe-Sort Attention, which uses a deterministic Z-order permutation to transform dense attention into static hardware-friendly sparse patterns, eliminating dynamic masking overhead. SparseSAM further introduces a (iii) Residual-Consistency MLP that routes only informative tokens through the MLP while propagating remaining tokens through the residual pathway. Across four segmentation benchmarks, SparseSAM loses only 0.004 mIoU at a 0.4 density and 0.021 mIoU at 0.3, a 2.10x reduction in accuracy loss versus token merging advances, while achieving 2x faster inference and 2.8x memory reduction.

Problem

• Segment Anything Model
• activation sparsification
• inference latency
• memory consumption
• model compression

Innovation

• structured sparsification
• Stripe-Sort Attention
• Residual-Consistency MLP
• training-free acceleration
• sparse activation