EdgeTAM: On-Device Track Anything Model

2025-01-13

AI Summary

To address SAM 2’s insufficient real-time performance for mobile video segmentation, this work proposes a lightweight deployment framework tailored for smartphones. We first identify the memory attention module as the primary latency bottleneck and introduce a novel 2D Spatial Perceiver architecture to efficiently model spatial relationships. A dual-granularity (global/local) query grouping mechanism is incorporated, alongside a zero-overhead memory distillation pipeline that jointly enables knowledge transfer and computational compression. Integrating a lightweight Transformer with on-device optimization techniques, our method achieves 16 FPS inference speed on an iPhone 15 Pro Max. Quantitatively, it attains J&F scores of 87.7 on DAVIS 2017, 70.0 on MOSE, and 72.3/71.7 on SA-V val/test—significantly outperforming existing mobile solutions and marking the first real-time deployment of a high-performance video segmentation model on smartphones.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

On top of Segment Anything Model (SAM), SAM 2 further extends its capability from image to video inputs through a memory bank mechanism and obtains a remarkable performance compared with previous methods, making it a foundation model for video segmentation task. In this paper, we aim at making SAM 2 much more efficient so that it even runs on mobile devices while maintaining a comparable performance. Despite several works optimizing SAM for better efficiency, we find they are not sufficient for SAM 2 because they all focus on compressing the image encoder, while our benchmark shows that the newly introduced memory attention blocks are also the latency bottleneck. Given this observation, we propose EdgeTAM, which leverages a novel 2D Spatial Perceiver to reduce the computational cost. In particular, the proposed 2D Spatial Perceiver encodes the densely stored frame-level memories with a lightweight Transformer that contains a fixed set of learnable queries. Given that video segmentation is a dense prediction task, we find preserving the spatial structure of the memories is essential so that the queries are split into global-level and patch-level groups. We also propose a distillation pipeline that further improves the performance without inference overhead. As a result, EdgeTAM achieves 87.7, 70.0, 72.3, and 71.7 J&F on DAVIS 2017, MOSE, SA-V val, and SA-V test, while running at 16 FPS on iPhone 15 Pro Max.

Problem

• SAM 2 Model
• Video Processing Efficiency
• Mobile Device Performance

Innovation

• EdgeTAM
• Lightweight Transformer
• Memory Optimization