On the Anisotropy of Score-Based Generative Models

2025-10-26

AI Summary

This work investigates how neural network architecture shapes the inductive biases of score-based generative models. We introduce Score Anisotropy Directions (SADs), a novel geometric analysis tool that quantifies directional anisotropy in the model’s output space, thereby establishing the first interpretable link between architectural choices and directional bias. Evaluated on synthetic data and standard image benchmarks (e.g., CIFAR-10, CelebA) using Wasserstein distance, SADs demonstrate strong predictive power: they reliably forecast downstream generation quality *prior to training* and correlate highly with actual sampling performance (Pearson’s *r* > 0.85). Crucially, SADs require only forward passes through untrained networks—no full training is needed—enabling efficient architecture selection and diagnostic analysis. Our approach advances theoretical interpretability and facilitates principled, controllable design of score-based generative models.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We investigate the role of network architecture in shaping the inductive biases of modern score-based generative models. To this end, we introduce the Score Anisotropy Directions (SADs), architecture-dependent directions that reveal how different networks preferentially capture data structure. Our analysis shows that SADs form adaptive bases aligned with the architecture's output geometry, providing a principled way to predict generalization ability in score models prior to training. Through both synthetic data and standard image benchmarks, we demonstrate that SADs reliably capture fine-grained model behavior and correlate with downstream performance, as measured by Wasserstein metrics. Our work offers a new lens for explaining and predicting directional biases of generative models.

Problem

• Investigating how network architecture affects score-based generative models' inductive biases
• Introducing Score Anisotropy Directions to reveal architecture-dependent data structure capture
• Predicting generalization ability and directional biases in score models before training

Innovation

• SADs reveal architecture-dependent data structure capture
• SADs form adaptive bases aligned with output geometry
• SADs predict generalization ability before training