EGA: Adapting Frozen Encoders for Vector Search with Bounded Out-of-Distribution Degradation

2026-05-07

AI Summary

This work addresses the severe out-of-distribution (OOD) performance degradation of frozen vision encoders when adapting to unseen class queries during deployment, a problem exacerbated by standard adapter training. To mitigate this, the authors propose Euclidean Geodesic Alignment (EGA), a novel approach that integrates residual adapters, zero initialization, local triplet loss, and hyperspherical projection. A key innovation is the introduction of a self-limiting dynamic mechanism that automatically halts adapter updates once local geometric structures are sufficiently aligned, thereby bounding OOD perturbations. Experiments demonstrate that EGA significantly improves worst-case label accuracy across five OOD benchmarks, with 96.5% of triplets requiring no gradient updates at convergence. Moreover, the method generalizes effectively beyond CLIP to stronger backbone architectures.

Abstract

Vector search systems built on frozen vision encoders face queries from unseen classes at deployment, yet existing adapter training collapses under this shift: high-capacity adapters with global contrastive losses silently reassign unseen-class samples to wrong seen-class clusters, dropping worst-case Label Precision by over 40 points below the frozen baseline in our tests. We propose Euclidean Geodesic Alignment (EGA), a residual adapter that couples three principles: zero initialization, local triplet loss, and hypersphere projection. These collectively induce a self-limiting dynamic: triplets that already satisfy a small margin stop producing gradients, so the adapter automatically stops updating where the local geometry is already correct. Our experiments show that at convergence $96.5\%$ of triplets are gradient-free, leaving unseen-class regions largely untouched while still enabling full-capacity refinement of seen classes. Across five diverse out-of-distribution (OOD) benchmarks, EGA achieves the highest worst-case Label Precision on the four primary splits and a consistent improvement on the fifth. The design also transfers to stronger backbones in addition to CLIP, and we provide an analytical justification linking gradient sparsity to bounded OOD perturbation.

Problem

• vector search
• frozen encoders
• out-of-distribution
• adapter training
• label precision

Innovation

• Euclidean Geodesic Alignment
• frozen encoder adaptation
• out-of-distribution robustness
• local triplet loss
• gradient sparsity