360° Image Perception with MLLMs: A Comprehensive Benchmark and a Training-Free Method

2026-03-17

AI Summary

This work addresses the challenges faced by multimodal large language models (MLLMs) in understanding 360° images—particularly geometric distortions and complex spatial relationships—and the absence of a systematic evaluation benchmark. To this end, the authors introduce 360Bench, the first comprehensive benchmark for 360° image understanding, comprising 7K high-resolution panoramic images and seven human-annotated subtasks. They further propose Free360, a training-free, modular reasoning framework that significantly enhances MLLM performance on 360° visual question answering through scene graph–guided decomposed reasoning, adaptive spherical transformation, and multi-view information fusion. Free360 offers an efficient, plug-and-play solution for panoramic image understanding without requiring model retraining.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multimodal Large Language Models (MLLMs) have shown impressive abilities in understanding and reasoning over conventional images. However, their perception of 360° images remains largely underexplored. Unlike conventional images, 360° images capture the entire surrounding environment, enabling holistic spatial reasoning but introducing challenges such as geometric distortion and complex spatial relations. To comprehensively assess MLLMs' capabilities to perceive 360° images, we introduce 360Bench, a Visual Question Answering (VQA) benchmark featuring 7K-resolution 360° images, seven representative (sub)tasks with annotations carefully curated by human annotators. Using 360Bench, we systematically evaluate seven MLLMs and six enhancement methods, revealing their shortcomings in 360° image perception. To address these challenges, we propose Free360, a training-free scene-graph-based framework for high-resolution 360° VQA. Free360 decomposes the reasoning process into modular steps, applies adaptive spherical image transformations to 360° images tailored to each step, and seamlessly integrates the resulting information into a unified graph representation for answer generation. Experiments show that Free360 consistently improves its base MLLM and provides a strong training-free solution for 360° VQA tasks. The source code and dataset will be publicly released upon acceptance.

Problem

• 360° image perception
• Multimodal Large Language Models
• geometric distortion
• spatial reasoning
• Visual Question Answering

Innovation

• 360-degree image perception
• training-free method
• scene graph
• spherical image transformation
• multimodal large language models