G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification

2026-08-27

AI Summary

本文提出G2D框架，通过生成式模型验证CLIP检索的候选标签，解决零样本图像分类中的标签召回与消歧问题。

Abstract

Zero-shot classification needs efficient label retrieval and fine-grained visual reasoning, yet discriminative and generative vision-language models fail in complementary ways.When CLIP's top-1 prediction is wrong, the correct label often remains in its top-$K$ shortlist, making disambiguation rather than recall the key challenge.Standalone generative models, however, are hindered by large label spaces and unconstrained outputs.This complementarity motivates separating broad candidate retrieval from fine-grained, image-grounded verification.We propose G2D, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.Candidate names and CLIP probabilities provide a structured prior for resolving visually similar classes.Fixed confidence routing, entropy-adaptive candidate sizing, and trie-constrained decoding focus generative reasoning on uncertain samples and ensure one valid output for each input at test time.Across eight benchmarks, G2D achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.Across seven generator configurations, candidate-set verification improves average accuracy by 1.08--27.42 percentage points.G2D also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning. Code: https://github.com/Harzva/G2D

Problem

• Zero-shot classification
• Label retrieval
• Fine-grained visual reasoning
• Generative models
• Discriminative models

Innovation

• Generative-to-Discriminative
• Zero-Shot Image Classification
• Candidate Verification
• Confidence Routing
• Trie-Constrained Decoding