Beyond Compression: Quantifying Spectral Accessibility in Vision Representations

2026-06-02

AI Summary

This study investigates the impact of projection transformations on the spectral structure of visual representations in vision-language models, disentangling this effect from the confounding influence of dimensional compression. To this end, the authors propose Residual Spectral Loss (RSL) to quantify the linear recoverability of band-limited Fourier energy and evaluate it through Fourier spectral analysis, random projection baselines, and frozen pretrained models (CLIP and DINOv2). The findings reveal that spectral accessibility varies non-monotonically with network depth, peaking at intermediate layers. While CLIP’s projection is approximately spectrally neutral, DINOv2’s [CLS] pooling induces structured high-frequency loss. These results highlight intermediate-layer representations and pooling mechanisms as key drivers of spectral transformation in such models.

Abstract

Vision-language models map visual features into a shared embedding space through learned projection layers, yet it remains unclear how these transformations alter the structure of visual information. This study examines changes in representation through spatial-frequency accessibility, measured by the linear recoverability of band-limited Fourier energy from model representations. To isolate effects beyond dimensionality reduction, we introduce Residual Spectral Loss (RSL), which evaluates changes relative to a dimension-matched random projection baseline. To reduce confounding effects from optimization, the analysis uses pretrained models with all parameters frozen. The experimental results show consistent frequency-dependent changes in accessibility across CLIP and DINOv2 on ImageNet and MS-COCO datasets. Spectral accessibility follows a non-monotonic trajectory across depth, peaking at intermediate layers before decreasing toward the output representation. The final transformation differs across architectures: CLIP's learned projection is spectrally neutral, with changes explained by compression, whereas DINOv2's [CLS] pooling induces a structured loss across the spectrum. These findings identify intermediate layers and pooling mechanisms as primary drivers of spectral transformation in modern vision encoders.

Problem

• spectral accessibility
• vision representations
• spatial-frequency
• embedding transformation
• frequency-dependent changes

Innovation

• Residual Spectral Loss
• spectral accessibility
• vision-language models
• frequency analysis
• representation learning