Blocked Gibbs meets Diffusion Transformers: Unsupervised Learning for Constraint Optimization

2026-05-24

AI Summary

Existing diffusion models struggle with constrained optimization problems involving general discrete variables or requiring global reasoning. This work proposes BloGDiT, the first method to integrate Blocked Gibbs sampling into diffusion models, enabling large-scale directed variable editing for constraint satisfaction and optimization within a Diffusion Transformer architecture. By combining block-wise Gaussian denoising, iterative resampling, and a block-size annealing schedule, BloGDiT introduces a powerful inductive bias that operates without supervision. The approach achieves state-of-the-art or competitive performance on diverse tasks—including Sudoku, graph coloring, maximum independent set, and MaxCut—demonstrating its effectiveness and generalization capability in generic discrete constrained optimization.

Abstract

Diffusion models have shown promise in learning to solve constraint optimization problems. However, they are mostly restricted to problems with binary variables and rely on graph neural networks, hindering their application to a broader range of problems such as those with general discrete variables or constraint structures that necessitate global rather than local reasoning. We investigate the use of Diffusion Transformers to address the aforementioned limitations. A naive implementation performs poorly due to a fundamental mismatch between the standard diffusion process and constraint solving: while the former applies small, incremental denoising across all variables, the latter requires substantially altering specific subsets of variables to attain feasibility or optimality. Our method, Blocked Gibbs Diffusion Transformer (BloGDiT), is the first to address this limitation by replacing standard joint Gaussian denoising with blocked Gaussian denoising. BloGDiT uses iterative block resampling and anneals the block size over time to facilitate large, targeted edits within a block of variables. Across Sudoku, Graph Coloring, Maximum Independent Set, and MaxCut, BloGDiT matches or outperforms existing methods, demonstrating that blocked Gibbs-style diffusion provides a highly effective inductive bias for Transformer-based constraint satisfaction and optimization.

Problem

• constraint optimization
• diffusion models
• discrete variables
• global reasoning
• unsupervised learning

Innovation

• Blocked Gibbs Sampling
• Diffusion Transformers
• Constraint Optimization
• Discrete Variables
• Inductive Bias