SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking

2026-06-23

AI Summary

This work addresses the tracking drift in SAM2 caused by its sole reliance on confidence scores for mask selection under occlusion, rapid motion, or distractors. The authors propose SENTRY, a training-free, plug-and-play pre-validation module for memory updates. Its key innovation lies in introducing temporal consistency verification at the memory-write stage, leveraging short-term trajectory backtracking and neighborhood-aware cycle-consistent matching to select spatiotemporally geometrically coherent segmentation masks. Without modifying the base architecture or requiring retraining, SENTRY substantially enhances robustness. The study also establishes the first unified, full-scale evaluation benchmark for SAM2-based tracking, achieving zero-shot state-of-the-art performance across nine datasets—including LaSOT and GOT-10k—with the SAM2-L variant running at 32.8 FPS on an A100 GPU while consuming only an additional 0.4–0.6 GB of GPU memory.

Abstract

We revisit the memory update mechanism in SAM2-based visual object tracking and identify confidence-only mask selection as the dominant cause of drift under occlusion, rapid motion, and distractors. We introduce SENTRY, a training-free, plug-and-play, refine-before-write module that validates each memory update for short-horizon temporal consistency before committing it. SENTRY aggregates diverse segmentation hypotheses per frame, backtracks them into short tracklets, and uses neighbor-aware cycle-consistent matching against recent trajectories to favor temporally and geometrically consistent masks. It leaves the base architecture untouched, replacing confidence-driven writes with consistency-validated ones. For fair evaluation, we re-evaluate major open-source SAM2-based trackers across all available scales and datasets, filling gaps in prior reports. Integrated into five strong baselines, SENTRY delivers consistent gains across nine benchmarks, achieving new zero-shot SOTA on LaSOT, LaSOT_ext, GOT-10k, VOT20, VOT22, and DiDi. Despite these checks, the SAM2-L version runs at 32.8 FPS on an A100, and across compatible hosts adds only about 0.4--0.6 GB VRAM. Our results provide the first unified all-scale evaluation of SAM2-based trackers and show that enforcing temporal validity at write time stabilizes memory-augmented tracking without retraining.

Problem

• visual tracking
• memory update
• tracking drift
• temporal consistency
• occlusion

Innovation

• temporal consistency
• memory validation
• neighbor-aware matching
• zero-shot tracking
• SAM2-based tracking