Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings

2026-06-13

AI Summary

This work addresses the limitation of conventional visual encoders in retrieval tasks, which rely solely on scalar distance supervision from class labels and thus struggle to capture fine-grained semantic differences between image pairs. To overcome this, the authors propose SAGA, a framework that leverages a frozen multimodal large language model (MLLM) to perform attribute-level semantic judgments on image pairs and converts these predictions into fine-grained supervision signals to guide the visual encoder toward learning attribute-aware embeddings—without requiring MLLM fine-tuning. By integrating Group Relative Policy Optimization (GRPO), attention distillation, and metric learning losses, SAGA steers the encoder to focus on salient attribute features. Experiments demonstrate that SAGA achieves 3–6 percentage point gains in zero-shot retrieval Recall@1 on CUB-200-2011, Cars-196, FGVC-Aircraft, and iNaturalist Aves, substantially outperforming existing approaches.

Abstract

Vision encoders for retrieval are typically trained with class-label supervision: each training pair reduces to a scalar that uniformly pushes the embedding apart or pulls it together, as if every visual attribute either differed or matched. A multimodal large language model (MLLM), shown the same pair, can articulate those attributes and use them to predict whether the images share a class. We propose \textbf{SAGA}, a framework that turns this language-grounded, attribute-aware perception into a training signal for the encoder itself. Specifically, we use Group Relative Policy Optimization (GRPO) to reward the MLLM for correct predictions on the vision encoder's tokens. Since correct predictions require those tokens to expose the specific attributes that differ or match between the pair, the gradient pushes the encoder to encode them, replacing the uniform pair-level scalar with attribute-resolved supervision. An auxiliary attention-distillation loss anchors the encoder's embedding to tokens the MLLM attended to, and a standard metric-learning loss shapes the embedding geometry for nearest-neighbour retrieval. The MLLM is frozen throughout and discarded at inference, matching the deployment cost of a metric-learning baseline. SAGA improves Recall@1 by 3 to 6 points over state-of-the-art baselines on CUB-200-2011, Cars-196, FGVC-Aircraft, and iNaturalist Aves on zero-shot image retrieval.

Problem

• visual embeddings
• semantic attributes
• metric learning
• image retrieval
• multimodal large language models

Innovation

• Semantic Attribute Gradients
• Frozen MLLMs
• Group Relative Policy Optimization
• Attention Distillation
• Zero-shot Image Retrieval