How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation

2026-07-04

AI Summary

This work addresses the opacity and poorly understood bias characteristics of diffusion-based classifiers in zero-shot classification. The authors introduce ASOB-Bench, the first benchmark designed to systematically evaluate decision biases along three dimensions: attribute binding, size-order bias, and background dependence. By integrating reconstruction error heatmaps and U-Net cross-attention visualizations, they uncover the underlying mechanisms driving these biases. Experimental results reveal that while diffusion classifiers exhibit fewer attribute mismatches compared to OpenCLIP, they suffer from more pronounced size-order bias and background dependence, leading to a significant drop in accuracy on ImageNet-B. These findings highlight distinct bias profiles relative to vision-language models and expose potential failure modes inherent to generative classification paradigms.

Abstract

Diffusion models have recently been repurposed for zero-shot classification, giving rise to diffusion classifiers that identify the best-matching text prompt by minimizing the noise-prediction error. Despite their growing adoption, how these models make classification decisions remains poorly understood. We introduce ASOB-Bench, a bias evaluation for diffusion classifiers along three dimensions: Attribute binding, Size-Order bias, and Background dependency. These dimensions serve not as an exhaustive taxonomy but as targeted probes of how the text-conditioned reconstruction-error score reaches a decision. Such a perspective is well studied for discriminative vision-language models, yet remains overlooked for diffusion classifiers. Extending an existing framework with five new attribute categories on newly constructed datasets, we find diffusion classifiers are less prone to attribute misbinding than an OpenCLIP baseline; on the established ComCo benchmark they are substantially more susceptible to size-order shortcuts; and on ImageNet-B they suffer far larger accuracy drops, revealing heavy reliance on background over foreground cues. Reconstruction-error heatmaps and U-Net cross-attention visualizations expose the mechanism behind each bias. Because diffusion classifiers share the same denoiser as text-to-image models, these single-pass diagnostics also point toward analogous failure modes in generation. Overall, diffusion classifiers exhibit a distinct bias profile from vision-language models, offering guidance for building more robust diffusion-based models.

Problem

• diffusion classifiers
• bias evaluation
• attribute binding
• size-order bias
• background dependency

Innovation

• diffusion classifiers
• bias evaluation
• attribute binding
• reconstruction-error analysis
• cross-attention visualization