SUMO: Segment and Track Any Motion with Nonlinear State Space Models

2026-06-29

AI Summary

This work addresses the limitations of existing visual object tracking and moving object segmentation methods in complex nonlinear motion scenarios, which often stem from their overreliance on visual cues. To overcome this, the authors propose SUMO, a novel framework that uniquely integrates robotics-inspired nonlinear state-space modeling with zero-shot visual segmentation, enabling unified tracking and segmentation without any task-specific training. SUMO introduces a selective unscented filter and a memory-frame reliability assessment mechanism to dynamically fuse predictions from multiple sources. Experimental results demonstrate that SUMO achieves state-of-the-art performance on both tasks, significantly enhancing modeling capacity and robustness in handling complex motion dynamics.

Abstract

Visual Object Tracking (VOT) and Moving Object Segmentation (MOS) are two fundamental tasks in computer vision that involve both spatial and temporal object dynamics. Existing methods rely predominantly on visual cues and thus often falter in real-world scenarios where object motions are inherently complex and nonlinear. To address this limitation, we propose SUMO, a zero-shot, training-free, unified framework integrating nonlinear dynamics with vision-based segmentation for accurate and consistent VOT and MOS. Specifically, we develop a nonlinear State Space Model (SSM) inspired by robotics principles to capture the complex object dynamics. Building on this model, we propose a Selective Unscented Filter (SUF) for accurate state estimation, which features a joint scoring mechanism and dynamically fuses multi-source predictions to identify the most plausible object state over time. Furthermore, we apply a memory selection mechanism to evaluate the reliability of memory frames. Our extensive experimental results show that SUMO achieves state-of-the-art performance on both VOT and MOS tasks.

Problem

• Visual Object Tracking
• Moving Object Segmentation
• Nonlinear Dynamics
• State Space Model

Innovation

• Nonlinear State Space Model
• Selective Unscented Filter
• Zero-shot Tracking
• Moving Object Segmentation
• Memory Selection Mechanism