SAP: Segment Any 4K Panorama

2026-03-13

AI Summary

Existing foundation models exhibit significant performance degradation on high-resolution 360° panoramic images, limiting their applicability in embodied intelligence and augmented reality. To address this, we propose a trajectory-aligned paradigm for panoramic instance segmentation, reformulating the task as overlapping perspective video segmentation sampled along fixed spherical trajectories. This approach preserves native 4K resolution while ensuring cross-view consistent instance propagation. Leveraging the InfiniGen engine, we construct a large-scale synthetic dataset comprising 183,440 annotated 4K panoramic images and adapt SAM2 with a memory-aligned video segmentation architecture trained under our trajectory-aligned framework. Evaluated on real-world 4K panoramic benchmarks, our method achieves a zero-shot mIoU improvement of 17.2 percentage points over the original SAM2, substantially outperforming current state-of-the-art approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Promptable instance segmentation is widely adopted in embodied and AR systems, yet the performance of foundation models trained on perspective imagery often degrades on 360° panoramas. In this paper, we introduce Segment Any 4K Panorama (SAP), a foundation model for 4K high-resolution panoramic instance-level segmentation. We reformulate panoramic segmentation as fixed-trajectory perspective video segmentation, decomposing a panorama into overlapping perspective patches sampled along a continuous spherical traversal. This memory-aligned reformulation preserves native 4K resolution while restoring the smooth viewpoint transitions required for stable cross-view propagation. To enable large-scale supervision, we synthesize 183,440 4K-resolution panoramic images with instance segmentation labels using the InfiniGen engine. Trained under this trajectory-aligned paradigm, SAP generalizes effectively to real-world 360° images, achieving +17.2 zero-shot mIoU gain over vanilla SAM2 of different sizes on real-world 4K panorama benchmark.

Problem

• panoramic segmentation
• instance segmentation
• 4K resolution
• 360-degree images
• foundation models

Innovation

• panoramic segmentation
• 4K resolution
• trajectory-aligned reformulation
• promptable instance segmentation
• foundation model