Performance Analysis and Optimization of 3D Generative Diffusion Models across GPU Architectures

2026-06-10

AI Summary

This work addresses the high GPU overhead and low inference efficiency of 3D medical diffusion models, such as Med-DDPM, in high-fidelity MRI synthesis—primarily caused by numerous U-Net evaluations and heterogeneous kernel behaviors. For the first time, it systematically characterizes their microarchitectural performance across three generations of NVIDIA GPUs, revealing that cuDNN convolutions and implicit GEMM kernels dominate training and identifying key bottlenecks in memory access patterns, tensor layouts, and Tensor Core utilization. Building on these insights, the study proposes architecture-aware optimizations: enabling TF32 Tensor Cores and adopting a 3D channels-last memory layout. Evaluated on an A100 GPU, this approach preserves generation quality while reducing SM cycles and dynamic instruction counts by up to 100×, boosting Tensor Core utilization from 1.45× to 9.98× and improving IPC by 7%.

Abstract

Diffusion models have become essential for high-fidelity 3D MRI synthesis, yet their deployment remains constrained by substantial GPU resource demands arising from hundreds of U-Net evaluations per sample and a highly heterogeneous kernel behavior. This paper performs a comprehensive performance analysis of the state-of-the-art medical diffusion model, Med-DDPM, across three generations of NVIDIA architectures to study kernel-level runtime breakdowns, instruction-mix characteristics, memory system utilization, warp-level activities, and profiler priority-score estimates. We show that training is overwhelmingly dominated by cuDNN convolution and implicit-GEMM kernels, with inefficiencies arising from memory-access patterns, tensor-layout conversions, and limited Tensor Core utilization. Guided by these insights, we evaluate two architecture-aware optimizations TF32 Tensor Core activation and a 3D channels-last layout and demonstrate that they reduce SM cycles by up to 100x, cut dynamic instructions by 100x, raise Tensor Core utilization from 1.45 to 9.98x, and increase IPC by 7% on A100, all without degrading synthesis quality.

Problem

• 3D generative diffusion models
• GPU resource demands
• U-Net evaluations
• heterogeneous kernel behavior
• 3D MRI synthesis

Innovation

• 3D diffusion models
• GPU performance analysis
• Tensor Core optimization
• channels-last layout
• medical image synthesis