Monge-Kantorovich Fitting With Sobolev Budgets

2024-09-25

arXiv.org

AI Summary

This work addresses the problem of Wasserstein-$p$ optimal approximation of high-dimensional probability measures by measures supported on low-dimensional subspaces, with Sobolev norms ($W^{k,q}$) explicitly constraining the complexity of transport maps—thereby unifying manifold learning and regularized generative modeling. Methodologically, it innovatively incorporates Sobolev norms into an optimal transport–driven dimensionality reduction framework, interpreting them as a “complexity budget” for transport maps. It introduces the barycenter field to characterize the gradient of the objective functional and establishes a near-strict monotonicity theory for the objective under higher-order differentiability assumptions. Furthermore, it constructs a natural discretization scheme with provable discrete consistency. The contributions provide a novel theoretical foundation for the role of regularization in generative models and extend the applicability of optimal transport to structured, regularized dimensionality reduction.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Given $m<n$, we consider the problem of ``best'' approximating an $n ext{-d}$ probability measure $ ho$ via an $m ext{-d}$ measure $ u$ such that $mathrm{supp} u$ has bounded total ``complexity.'' When $ ho$ is concentrated near an $m ext{-d}$ set we may interpret this as a manifold learning problem with noisy data. However, we do not restrict our analysis to this case, as the more general formulation has broader applications. We quantify $ u$'s performance in approximating $ ho$ via the Monge-Kantorovich (also called Wasserstein) $p$-cost $mathbb{W}_p^p( ho, u)$, and constrain the complexity by requiring $mathrm{supp} u$ to be coverable by an $f : mathbb{R}^{m} o mathbb{R}^{n}$ whose $W^{k,q}$ Sobolev norm is bounded by $ell geq 0$. This allows us to reformulate the problem as minimizing a functional $mathscr J_p(f)$ under the Sobolev ``budget'' $ell$. This problem is closely related to (but distinct from) principal curves with length constraints when $m=1, k = 1$ and an unsupervised analogue of smoothing splines when $k>1$. New challenges arise from the higher-order differentiability condition. We study the ``gradient'' of $mathscr J_p$, which is given by a certain vector field that we call the barycenter field, and use it to prove a nontrivial (almost) strict monotonicity result. We also provide a natural discretization scheme and establish its consistency. We use this scheme as a toy model for a generative learning task, and by analogy, propose novel interpretations for the role regularization plays in improving training.

Problem

• Approximate high-dimensional probability measures with lower-dimensional ones
• Optimize approximation using Monge-Kantorovich cost under Sobolev constraints
• Study gradient behavior and propose discretization for generative learning

Innovation

• Uses Monge-Kantorovich fitting for approximation
• Applies Sobolev budgets to constrain complexity
• Introduces barycenter field for gradient analysis