Learning to Align Generative Appearance Priors for Fine-grained Image Retrieval

2026-05-10

AI Summary

Existing fine-grained image retrieval methods rely heavily on supervised signals from seen categories and prioritize semantic features, which limits their generalization to unseen categories. To address this limitation, this work proposes the GAPan network, which shifts the learning objective from category prediction to appearance modeling. GAPan uniquely integrates invertible normalizing flows with a class-conditional Gaussian prior, enabling forward density estimation and reverse sampling to generate appearance-aware anchor points. Furthermore, it introduces a prior-driven alignment mechanism that preserves fine-grained appearance details across categories. The proposed method achieves state-of-the-art performance on multiple fine-grained and general image retrieval benchmarks, significantly enhancing generalization in zero-shot and cross-category retrieval scenarios.

Abstract

Fine-grained image retrieval (FGIR) typically relies on supervision from seen categories to learn discriminative embeddings for retrieving unseen categories. However, such supervision often biases retrieval models toward the semantics of seen categories rather than the underlying appearance characteristics that generalize across categories, thereby limiting retrieval performance on unseen categories. To tackle this, we propose GAPan, a Generative Appearance Prior alignment network that reformulates the learning objective from category prediction toward appearance modeling. Technically, GAPan treats retrieval features with an invertible density model based on normalizing flows. In the forward direction, the flow maps all instance features into a latent density space, where each seen category is modeled by a class-conditional Gaussian prior and optimized via exact likelihood estimation. This formulation preserves richer appearance details by leveraging the invertible property of the flows. In the reverse direction, samples from the high-density regions of these learned priors are mapped back to the feature space to produce appearance-aware anchors that reflect intra-category variation. These anchors supervise a prior-driven alignment objective that aligns retrieval embeddings with category-specific appearance distributions, thereby improving generalization to unseen categories. Evaluations demonstrate that our GAPan achieves state-of-the-art performance on both widely-used fine- and coarse-grained benchmarks.

Problem

• fine-grained image retrieval
• unseen categories
• appearance generalization
• supervision bias
• category semantics

Innovation

• Generative Appearance Prior
• Normalizing Flows
• Fine-grained Image Retrieval
• Prior-driven Alignment
• Invertible Density Model