RayTun3R: Online Camera Adaptation in 3D Foundation Models

2026-07-02

AI Summary

This work addresses the significant performance degradation of existing 3D foundation models on fisheye camera images, which stems from positional encodings that implicitly assume pinhole camera geometry. To overcome this limitation, the authors propose a lightweight online adaptation method that operates with the pretrained model frozen. The approach employs a parameter-efficient residual correction mechanism to adjust positional encodings, combined with a parameter-free tokenization strategy and a geometry-aware loss to refine mesh coordinates, thereby fully eliminating pinhole bias. Notably, the method requires no multi-view reasoning and achieves a 2–12× reduction in rotation error on fisheye datasets with field-of-view angles ranging from 110° to 200°, while preserving depth estimation accuracy. It uses only approximately 1/14 the parameters of LoRA.

Abstract

Recent 3D foundation models, such as DUSt3R, MASt3R, VGGT, $π^3$, and Depth Anything 3, provide strong feed-forward depth and pose estimates on pinhole imagery, but degrade sharply under fisheye camera geometry. We show that this failure is partly caused by a pinhole camera bias in the positional encodings of pretrained 3D foundation models, and propose RayTun3R, a lightweight camera adaptation approach. It keeps the pretrained network fixed and adapts only lightweight components tied to token position and camera geometry. RayTun3R learns parameter-efficient residual corrections to absolute and rotary positional encodings, together with parameter-free tokenization and corrections to prediction-grid coordinates that remove residual pinhole assumptions. The resulting adapter contains only 10,752 trainable parameters and can be learned from a short temporal segment using geometric losses. Once adapted, RayTun3R transfers effectively to the remaining frames of the sequence without incurring additional runtime costs. Across diverse fisheye datasets with fields of view from $110^\circ$ to $200^\circ$, our adapter reduces rotation error by $2$-$12\times$ relative to the unadapted model, outperforms LoRA while using $\sim\!14\times$ fewer trainable parameters, improves pose over adaptation-free baselines while avoiding their multi-view inference cost, and remains competitive on depth accuracy.

Problem

• 3D foundation models
• fisheye camera
• positional encoding
• camera adaptation
• pinhole bias

Innovation

• camera adaptation
• 3D foundation models
• fisheye imagery
• positional encoding
• parameter-efficient tuning