Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360-Degree Videos

2025-08-27

AI Summary

Existing spherical saliency prediction methods suffer from low accuracy due to severe spherical distortion in 360° video and ineffective integration of spatial audio. To address this, we propose an audio-visual collaborative spherical saliency prediction model. Methodologically: (1) We design a spherical geometry-aware spatiotemporal attention mechanism to model gaze dynamics under continuous 360° viewing; (2) We introduce the first audio-conditioned Transformer adapter, enabling fine-grained modulation of visual features by spatial audio cues; (3) We construct YT360-EyeTracking—the first large-scale 360° eye-tracking dataset—comprising 120 hours of video and 24 million fixation points. Our model achieves state-of-the-art performance across multiple benchmarks, with an average AUC improvement of 4.2%, demonstrating the critical role of audio guidance and spherical geometric modeling in saliency prediction.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Omnidirectional videos (ODVs) are redefining viewer experiences in virtual reality (VR) by offering an unprecedented full field-of-view (FOV). This study extends the domain of saliency prediction to 360-degree environments, addressing the complexities of spherical distortion and the integration of spatial audio. Contextually, ODVs have transformed user experience by adding a spatial audio dimension that aligns sound direction with the viewer's perspective in spherical scenes. Motivated by the lack of comprehensive datasets for 360-degree audio-visual saliency prediction, our study curates YT360-EyeTracking, a new dataset of 81 ODVs, each observed under varying audio-visual conditions. Our goal is to explore how to utilize audio-visual cues to effectively predict visual saliency in 360-degree videos. Towards this aim, we propose two novel saliency prediction models: SalViT360, a vision-transformer-based framework for ODVs equipped with spherical geometry-aware spatio-temporal attention layers, and SalViT360-AV, which further incorporates transformer adapters conditioned on audio input. Our results on a number of benchmark datasets, including our YT360-EyeTracking, demonstrate that SalViT360 and SalViT360-AV significantly outperform existing methods in predicting viewer attention in 360-degree scenes. Interpreting these results, we suggest that integrating spatial audio cues in the model architecture is crucial for accurate saliency prediction in omnidirectional videos. Code and dataset will be available at https://cyberiada.github.io/SalViT360.

Problem

• Predicting visual saliency in 360-degree videos
• Addressing spherical distortion and audio integration challenges
• Leveraging audio-visual cues for accurate attention prediction

Innovation

• Spherical Vision Transformers with geometry-aware attention
• Transformer adapters conditioned on spatial audio input
• Novel dataset YT360-EyeTracking for audio-visual saliency