Learning normalized image densities via dual score matching

2025-06-05

AI Summary

This work addresses key challenges in high-dimensional image density modeling—namely, the curse of dimensionality, poor generalization, and energy inconsistency across noise scales. We propose Dual Score Matching, a novel framework that reparameterizes the score network to directly model a normalized energy function. It jointly optimizes two objectives: the input-space gradient (primary) and the noise-level gradient (auxiliary), ensuring both normalizability and consistency of the energy function across multi-scale noise perturbations. Theoretically, we show that local image manifold dimensionality and density dynamically adapt to semantic content—departing from conventional assumptions of fixed low-dimensional manifolds. Empirically, our method achieves state-of-the-art negative log-likelihood on ImageNet64. The resulting energy model exhibits strong generalization: log-probability estimates are nearly sample-independent. Extensive experiments further validate the content-adaptive nature of image density structure.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Learning probability models from data is at the heart of many machine learning endeavors, but is notoriously difficult due to the curse of dimensionality. We introduce a new framework for learning emph{normalized} energy (log probability) models that is inspired from diffusion generative models, which rely on networks optimized to estimate the score. We modify a score network architecture to compute an energy while preserving its inductive biases. The gradient of this energy network with respect to its input image is the score of the learned density, which can be optimized using a denoising objective. Importantly, the gradient with respect to the noise level provides an additional score that can be optimized with a novel secondary objective, ensuring consistent and normalized energies across noise levels. We train an energy network with this emph{dual} score matching objective on the ImageNet64 dataset, and obtain a cross-entropy (negative log likelihood) value comparable to the state of the art. We further validate our approach by showing that our energy model emph{strongly generalizes}: estimated log probabilities are nearly independent of the specific images in the training set. Finally, we demonstrate that both image probability and dimensionality of local neighborhoods vary significantly with image content, in contrast with traditional assumptions such as concentration of measure or support on a low-dimensional manifold.

Problem

• Learning normalized energy models for image densities
• Optimizing dual score matching for consistent energy normalization
• Validating generalization of energy models across image datasets

Innovation

• Modified score network for energy computation
• Dual score matching for normalized energies
• Denoising objective for gradient optimization