From Circuits to Dynamics: Understanding and Stabilizing Failure in 3D Diffusion Transformers

2026-02-11

AI Summary

This work addresses the “melting” phenomenon in 3D diffusion Transformers for sparse point cloud completion, where minor input perturbations lead to fragmented outputs. Through mechanistic interpretability, the failure is traced to aberrant cross-attention activations in early denoising stages, and diffusion dynamics analysis reveals an underlying symmetry-breaking bifurcation. The study establishes, for the first time, a circuit-level connection between attention mechanisms and diffusion trajectory dynamics, leading to PowerRemap—a test-time control method that requires no retraining. Extensive experiments across architectures (WaLa, Make-a-Shape) and datasets (GSO, SimJEB) demonstrate the prevalence of melting, while PowerRemap achieves up to 98.3% success in stabilizing generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reliable surface completion from sparse point clouds underpins many applications spanning content creation and robotics. While 3D diffusion transformers attain state-of-the-art results on this task, we uncover that they exhibit a catastrophic mode of failure: arbitrarily small on-surface perturbations to the input point cloud can fracture the output into multiple disconnected pieces -- a phenomenon we call Meltdown. Using activation-patching from mechanistic interpretability, we localize Meltdown to a single early denoising cross-attention activation. We find that the singular-value spectrum of this activation provides a scalar proxy: its spectral entropy rises when fragmentation occurs and returns to baseline when patched. Interpreted through diffusion dynamics, we show that this proxy tracks a symmetry-breaking bifurcation of the reverse process. Guided by this insight, we introduce PowerRemap, a test-time control that stabilizes sparse point-cloud conditioning. We demonstrate that Meltdown persists across state-of-the-art architectures (WaLa, Make-a-Shape), datasets (GSO, SimJEB) and denoising strategies (DDPM, DDIM), and that PowerRemap effectively counters this failure with stabilization rates of up to 98.3%. Overall, this work is a case study on how diffusion model behavior can be understood and guided based on mechanistic analysis, linking a circuit-level cross-attention mechanism to diffusion-dynamics accounts of trajectory bifurcations.

Problem

• 3D diffusion transformers
• surface completion
• catastrophic failure
• point cloud perturbation
• output fragmentation

Innovation

• diffusion transformers
• mechanistic interpretability
• spectral entropy
• symmetry-breaking bifurcation
• PowerRemap