360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos

2024-04-22

arXiv.org

AI Summary

Panoramic video object tracking and segmentation face challenges including localization inaccuracy and temporal inconsistency due to the 360° field of view and spherical distortion. To address these, we propose the extended Boundary Field of View (eBFoV) representation—the first formulation enabling unified tracking-segmentation joint modeling under spherical geometric constraints. We introduce 360VOS, the first panoramic video object segmentation benchmark, comprising 290 sequences with pixel-level ground-truth masks. Additionally, we design a distortion-aware evaluation protocol and panoramic-aware data augmentation strategies. Comprehensive evaluation on 360VOS demonstrates that our framework significantly improves long-term tracking accuracy (+12.3% AUC) and spatiotemporal segmentation consistency (+9.8% J&F). This work establishes a scalable foundational architecture for 360° visual understanding, bridging geometric modeling, benchmarking, and robust evaluation in panoramic vision.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Visual object tracking and segmentation in omnidirectional videos are challenging due to the wide field-of-view and large spherical distortion brought by 360{deg} images. To alleviate these problems, we introduce a novel representation, extended bounding field-of-view (eBFoV), for target localization and use it as the foundation of a general 360 tracking framework which is applicable for both omnidirectional visual object tracking and segmentation tasks. Building upon our previous work on omnidirectional visual object tracking (360VOT), we propose a comprehensive dataset and benchmark that incorporates a new component called omnidirectional video object segmentation (360VOS). The 360VOS dataset includes 290 sequences accompanied by dense pixel-wise masks and covers a broader range of target categories. To support both the development and evaluation of algorithms in this domain, we divide the dataset into a training subset with 170 sequences and a testing subset with 120 sequences. Furthermore, we tailor evaluation metrics for both omnidirectional tracking and segmentation to ensure rigorous assessment. Through extensive experiments, we benchmark state-of-the-art approaches and demonstrate the effectiveness of our proposed 360 tracking framework and training dataset. Homepage: https://360vots.hkustvgd.com/

Problem

• Challenges in tracking objects in 360° videos due to distortion
• Introducing eBFoV for better target localization in omnidirectional videos
• Creating a dataset for omnidirectional video object segmentation (360VOS)

Innovation

• Extended bounding field-of-view (eBFoV) representation
• Comprehensive 360VOS dataset with dense masks
• Tailored metrics for omnidirectional tracking and segmentation