Fourier-Attentive Representation Learning: A Fourier-Guided Framework for Few-Shot Generalization in Vision-Language Models

2025-12-03

AI Summary

To address the limited cross-domain generalization of vision-language models (VLMs) in few-shot learning—caused by entanglement between image structure and style features—this paper proposes a Fourier-domain disentanglement framework for representation learning. The method explicitly separates structural and stylistic information by modeling phase spectra (for structure) and magnitude spectra (for style), and introduces learnable query tokens that generate disentangled structure/style representations via dual cross-attention. Furthermore, an asymmetric frequency-domain feature injection strategy is designed to enhance vision–language alignment. The entire framework is jointly optimized end-to-end with mainstream large-scale pretrained VLMs. Extensive evaluation across 15 few-shot benchmark datasets demonstrates significant improvements over state-of-the-art baselines, particularly in cross-domain transfer tasks, where our approach exhibits superior robustness and generalization capability.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large-scale pre-trained Vision-Language Models (VLMs) have demonstrated strong few-shot learning capabilities. However, these methods typically learn holistic representations where an image's domain-invariant structure is implicitly entangled with its domain-specific style. This presents an opportunity to further enhance generalization by disentangling these visual cues. In this paper, we propose Fourier-Attentive Representation Learning (FARL), a novel framework that addresses this by explicitly disentangling visual representations using Fourier analysis. The core of our method is a dual cross-attention mechanism, where learnable representation tokens separately query an image's structural features (from the phase spectrum) and stylistic features (from the amplitude spectrum). This process yields enriched, disentangled tokens that are then injected deep into the VLM encoders to guide adaptation. Our design, which includes an asymmetric injection strategy, forces the model to learn a more robust vision-language alignment. Extensive experiments on 15 datasets demonstrate the effectiveness of our approach.

Problem

• Disentangles visual representations using Fourier analysis
• Separates structural and stylistic features in images
• Enhances few-shot generalization in vision-language models

Innovation

• Fourier analysis disentangles structural and stylistic features
• Dual cross-attention queries phase and amplitude spectra separately
• Asymmetric injection enforces robust vision-language alignment