OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning

2026-06-29

AI Summary

Existing panoramic image benchmarks are largely confined to simple tasks relying on local cues or single-step reasoning, making them inadequate for evaluating models’ capacity for global multi-hop spatial reasoning. To address this limitation, this work proposes OmniCoT—the first comprehensive evaluation and training framework tailored for global multi-hop spatial reasoning in panoramic images. It introduces a chain-of-thought annotation paradigm anchored by directional and proximity relationships and constructs a unified dataset comprising synthetic (OmniCoT-B), real-world (OmniCoT-Real), and instructional (OmniCoT-T) subsets. A two-stage training strategy combining supervised fine-tuning with geometry-consistent reinforcement learning (GRPO) explicitly aligns intermediate reasoning steps with panoramic evidence. The resulting OmniCoT-R1 model significantly outperforms existing approaches in both reasoning accuracy and logical coherence.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated promising spatial reasoning capabilities, while these abilities remain underexplored in the emerging visual modality of panoramic imagery. The full 360°$\times$180° field of view of panoramas essentially supports complex global multi-step reasoning, which is also the fundamental advantage of panoramas in applications such as embodied intelligence. However, existing panoramic benchmarks largely focus on simplistic queries that rely on local cues or single-/few-step reasoning, thereby ignoring the fundamental advantage of panoramas and failing to fully exploit their potential. To address this gap, we introduce OmniCoT, a panoramic spatial reasoning suite designed to enable MLLMs to use global evidence and perform multi-step inference across viewpoints. It includes OmniCoT-B (6.7K data) for evaluation, which measures both answer accuracy and reasoning quality, OmniCoT-Real (1K data) as a manually annotated real-world subset to quantify the Sim-to-Real gap. For training, OmniCoT-T (14.3K data) is purpose-built with structured stepwise Chain-of-Thought annotations that explicitly link intermediate reasoning steps to panoramic evidence. Based on OmniCoT-T, we introduce OmniCoT-R1 and adopt a two-stage training strategy tailored to the geometrically complex panoramic space, where Supervised Fine-tuning (SFT) anchors reasoning to panoramic evidence (e.g., bearings, proximity) and GRPO penalizes geometrically incoherent paths to consolidate global 360° spatial consistency. Through OmniCoT, we aim to recalibrate the difficulty of panoramic spatial reasoning to better align with the intrinsic capabilities of panoramic imagery, thereby fostering meaningful progress in this research area.

Problem

• panoramic reasoning
• multimodal large language models
• global spatial reasoning
• multi-step inference
• 360-degree imagery

Innovation

• panoramic reasoning
• multimodal large language models
• Chain-of-Thought
• global spatial consistency
• Sim-to-Real gap