Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance

2026-03-20

AI Summary

This work addresses the issue of gradient error accumulation in diffusion models during iterative generation, which arises from misalignment between training objectives and inference dynamics and ultimately impairs generalization. Building upon the “weak-to-strong” guidance principle, the study systematically characterizes the effective operating regimes of classifier-free guidance (CFG) and adaptive guidance (AG) for the first time, and introduces a segmented guidance (SGG) strategy. SGG enhances generation quality by blending guidance signals in distinct segments, jointly optimizing performance during both inference and training. Notably, SGG improves inference without requiring additional training and integrates seamlessly into architectures such as Stable Diffusion 3/3.5 and Transformers. Experiments demonstrate that SGG significantly boosts model generalization in both conditional and unconditional generation tasks, outperforming existing training-free guidance methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models generate synthetic images through an iterative refinement process. However, the misalignment between the simulation-free objective and the iterative process often causes accumulated gradient error along the sampling trajectory, which leads to unsatisfactory results and a failure to generalize. Guidance techniques like Classifier Free Guidance (CFG) and AutoGuidance (AG) alleviate this by extrapolating between the main and inferior signal for stronger generalization. Despite empirical success, the effective operational regimes of prevalent guidance methods are still under-explored, leading to ambiguity when selecting the appropriate guidance method given a precondition. In this work, we first conduct synthetic comparisons to isolate and demonstrate the effective regime of guidance methods represented by CFG and AG from the perspective of weak-to-strong principle. Based on this, we propose a hybrid instantiation called SGG under the principle, taking the benefits of both. Furthermore, we demonstrate that the W2S principle along with SGG can be migrated into the training objective, improving the generalization ability of unguided diffusion models. We validate our approach with comprehensive experiments. At inference time, evaluations on SD3 and SD3.5 confirm that SGG outperforms existing training-free guidance variants. Training-time experiments on transformer architectures demonstrate the effective migration and performance gains in both conditional and unconditional settings. Code is available at https://github.com/851695e35/SGG.

Problem

• diffusion models
• generalization
• guidance methods
• gradient error
• sampling trajectory

Innovation

• Weak-to-Strong Guidance
• Segmented Guidance (SGG)
• Diffusion Model Generalization
• Training-Free Guidance
• Gradient Error Mitigation