🤖 AI Summary
Existing post-training quantization methods overlook the coupled structure between the conditional and unconditional branches in Classifier-Free Guidance (CFG), leading to unconditional branch drift and degraded generation quality. This work is the first to identify and formally characterize this “branch drift trap” and proposes Guidance-Aware Mixed Precision (GAMP), a novel quantization strategy that calibrates models using the guidance-predicted output as the target. By evaluating per-layer activation sensitivity through guidance output degradation, GAMP employs a greedy knapsack-based approach to allocate mixed precision bit-widths. This mechanism inherently prevents branch drift, significantly enhancing generation fidelity while maintaining low-bit deployment efficiency. Moreover, it circumvents spurious optima arising from conventional calibration metrics, thereby aligning theoretical compression gains with actual inference performance.
📝 Abstract
Deploying classifier-free guidance (CFG) diffusion models under real-world compute budgets requires quantization, yet existing post-training quantization (PTQ) methods treat CFG models as single-branch networks, ignoring the paired conditional/unconditional structure that CFG inference fundamentally relies on. This structural blind spot has two consequences. At the system level, the two-pass CFG execution pattern imposes a latency overhead that parameter-count and bit-operation metrics conceal entirely, and commodity INT8 inference stacks fail to realize the theoretical efficiency gains that BOPs calculations promise. At the algorithmic level, calibrating against the guidance gap alone admits an exact null space: a quantized model can achieve perfect gap-fidelity diagnostics while the unconditional branch drifts arbitrarily, corrupting every guided prediction at inference time. This paper terms this the branch-drift trap, proves its existence analytically, and confirms it empirically through a false-positive result in which the best-calibrated model by standard diagnostics simultaneously produces the worst sample quality. To close the trap, Guidance-Aware Mixed Precision (GAMP) is proposed, which calibrates directly on the guided prediction, derives per-layer activation-bit sensitivity from guided-output degradation, and allocates bits via a greedy knapsack -- provably preventing unconditional branch drift by construction.