FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception

2026-04-11

AI Summary

This work addresses the limitations of existing vision foundation models, which rely on the pinhole camera assumption and struggle with the severe radial distortion inherent in fisheye images, further hindered by the scarcity of large-scale annotated data. The authors propose an efficient adaptation strategy that requires no fine-tuning of the backbone: they freeze a DINOv2 model and employ low-rank adaptation (LoRA) to transfer its self-supervised features, while introducing Fisheye Rotary Position Encoding (FishRoPE). FishRoPE reformulates the attention mechanism in spherical coordinates, computing similarity based on angular rather than pixel distance. This architecture-agnostic module incurs minimal computational overhead and naturally reduces to the standard positional encoding under pinhole assumptions. The method achieves state-of-the-art performance, attaining 54.3 mAP on WoodScape 2D object detection and 65.1 mIoU on SynWoodScapes BEV segmentation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

Vision foundation models (VFMs) and Bird's Eye View (BEV) representation have advanced visual perception substantially, yet their internal spatial representations assume the rectilinear geometry of pinhole cameras. Fisheye cameras, widely deployed on production autonomous vehicles for their surround-view coverage, exhibit severe radial distortion that renders these representations geometrically inconsistent. At the same time, the scarcity of large-scale fisheye annotations makes retraining foundation models from scratch impractical. We present \ours, a lightweight framework that adapts frozen VFMs to fisheye geometry through two components: a frozen DINOv2 backbone with Low-Rank Adaptation (LoRA) that transfers rich self-supervised features to fisheye without task-specific pretraining, and Fisheye Rotary Position Embedding (FishRoPE), which reparameterizes the attention mechanism in the spherical coordinates of the fisheye projection so that both self-attention and cross-attention operate on angular separation rather than pixel distance. FishRoPE is architecture-agnostic, introduces negligible computational overhead, and naturally reduces to the standard formulation under pinhole geometry. We evaluate \ours on WoodScape 2D detection (54.3 mAP) and SynWoodScapes BEV segmentation (65.1 mIoU), where it achieves state-of-the-art results on both benchmarks.

Problem

• fisheye cameras
• radial distortion
• vision foundation models
• geometric inconsistency
• Bird's Eye View

Innovation

• FishRoPE
• fisheye perception
• rotary position embedding
• LoRA
• BEV representation