Self-Improving Diffusion Classifiers with Minority Preference Optimization

2026-07-04

AI Summary

This work addresses the significant performance degradation of diffusion-based classifiers in low-density regions of the data manifold—typically corresponding to minority classes—due to a perceptual bias toward majority classes. The study establishes, for the first time, a direct link between sampling in minority regions and the classifier’s perceptual capability, and introduces a novel self-boosting zero-shot classification paradigm that requires neither additional images nor external models. By leveraging arbitrary textual descriptions to generate candidate samples and employing a reward mechanism to guide coverage of underrepresented regions, the method refines a pretrained diffusion model through LoRA fine-tuning and Group Relative Policy Optimization. Experiments across five standard benchmarks demonstrate substantial improvements in zero-shot recognition accuracy for minority classes.

Abstract

Prior studies have demonstrated that diffusion classifiers achieve robust zero-shot classification performance. However, their effectiveness is strongly tied to the pretraining data distribution: they perform well in majority, high-density regions of the data manifold, but are significantly less accurate in minority, low-density regions. Although prior works on minority sampling have focused on generating more minority-like images, what minority sampling fundamentally enables beyond generation remains underexplored. In this paper, we reveal a direct relationship between minority sampling in generation and the perception capability of diffusion classifiers. Specifically, we show that enhancing minority sampling broadens the coverage of underrepresented regions on the data manifold, thereby improving diffusion-based recognition. To exploit this connection, we propose \textit{Self-Improving Diffusion Classifiers with Minority Preference Optimization} (MiPO), which fine-tunes a pretrained diffusion model using minority preference rewards. Using only arbitrary caption data, MiPO generates candidate samples, rewards those that better cover minority regions, and optimizes the model with LoRA and Group Relative Policy Optimization, without additional image data, external foundation models, or external reward models. This enables stable, prompt-adaptive minority sampling and translates low-density generative coverage into improved zero-shot diffusion classification. To sum up, we show that diffusion classifier perception is biased toward majority regions, demonstrate that this bias can be alleviated through minority preference optimization, and evaluate MiPO on five standard datasets.

Problem

• diffusion classifiers
• minority regions
• data manifold
• zero-shot classification
• perception bias

Innovation

• diffusion classifier
• minority sampling
• preference optimization
• zero-shot classification
• data manifold coverage