Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics

2026-04-27

AI Summary

This study addresses the limited ability of existing CLIP models to comprehend the unique visual semantics of 360-degree panoramic images—particularly horizontal cyclic translation invariance—resulting in suboptimal text-image alignment. The work formally defines 360-degree vision-language semantics for the first time and introduces a keyword manipulation strategy alongside a horizontal cyclic translation evaluation protocol. To enhance the model’s sensitivity to cyclic structures, a LoRA-based fine-tuning framework is proposed. Experimental results demonstrate that while the original CLIP model can recognize 360-degree textual prompts, its visual semantic consistency remains weak; after fine-tuning, however, its 360-degree visual understanding improves significantly, validating the effectiveness of the proposed approach. The findings also reveal a trade-off between task-specific optimization and general-purpose semantic performance.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The dream of instantly creating rich 360-degree panoramic worlds from text is rapidly becoming a reality, yet a crucial gap exists in our ability to reliably evaluate their semantic alignment. Contrastive Language-Image Pre-training (CLIP) models, standard AI evaluators, predominantly trained on perspective image-text pairs, face an open question regarding their understanding of the unique characteristics of 360-degree panoramic image-text pairs. This paper addresses this gap by first introducing two concepts: \emph{360-degree textual semantics}, semantic information conveyed by explicit format identifiers, and \emph{360-degree visual semantics}, invariant semantics under horizontal circular shifts. To probe CLIP's comprehension of these semantics, we then propose novel evaluation methodologies using keyword manipulation and horizontal circular shifts of varying magnitudes. Rigorous statistical analyses across popular CLIP configurations reveal that: (1) CLIP models effectively leverage explicit textual identifiers, demonstrating an understanding of 360-degree textual semantics; and (2) CLIP models fail to robustly preserve semantic alignment under horizontal circular shifts, indicating limited comprehension of 360-degree visual semantics. To address this limitation, we propose a LoRA-based fine-tuning framework that explicitly instills invariance to circular shifts. Our fine-tuned models exhibit improved comprehension of 360-degree visual semantics, though with a slight degradation in original semantic evaluation performance, highlighting a fundamental trade-off in adapting CLIP to 360-degree panoramic images. Code is available at https://github.com/littlewhitesea/360Semantics.

Problem

• 360-degree semantics
• CLIP
• semantic alignment
• panoramic images
• circular shift invariance

Innovation

• 360-degree semantics
• CLIP evaluation
• circular shift invariance
• LoRA fine-tuning
• panoramic image-text alignment