Sigma Flows for Image and Data Labeling and Learning Structured Prediction

2024-08-28

arXiv.org

AI Summary

This work addresses structured label prediction on Riemannian manifolds, particularly for Euclidean-domain data such as images. We propose SigmaFlow—a Riemannian gradient flow driven by a generalized harmonic energy—that maps the input domain manifold to a statistical manifold endowed with the Fisher–Rao metric, enabling end-to-end structured prediction. Our key innovations include a **time-varying domain metric mechanism** and a **compact learnable parametrization**, jointly embedding dynamic geometric priors. By unifying harmonic mapping theory (governed by PDEs) with structured learning, we establish a geometrically consistent modeling paradigm. The model is grounded in nonlinear geometric PDEs, the Laplace–Beltrami operator, and information geometry, and is solved via geometric integration and manifold optimization. Extensive evaluation on multi-label image annotation tasks demonstrates superior expressivity and generalization. Furthermore, we uncover deep structural parallels between SigmaFlow and Transformers, offering a novel geometric foundation for scientific machine learning.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

This paper introduces the sigma flow model for the prediction of structured labelings of data observed on Riemannian manifolds, including Euclidean image domains as special case. The approach combines the Laplace-Beltrami framework for image denoising and enhancement, introduced by Sochen, Kimmel and Malladi about 25 years ago, and the assignment flow approach introduced and studied by the authors. The sigma flow arises as Riemannian gradient flow of generalized harmonic energies and thus is governed by a nonlinear geometric PDE which determines a harmonic map from a closed Riemannian domain manifold to a statistical manifold, equipped with the Fisher-Rao metric from information geometry. A specific ingredient of the sigma flow is the mutual dependency of the Riemannian metric of the domain manifold on the evolving state. This makes the approach amenable to machine learning in a specific way, by realizing this dependency through a mapping with compact time-variant parametrization that can be learned from data. Proof of concept experiments demonstrate the expressivity of the sigma flow model and prediction performance. Structural similarities to transformer network architectures and networks generated by the geometric integration of sigma flows are pointed out, which highlights the connection to deep learning and, conversely, may stimulate the use of geometric design principles for structured prediction in other areas of scientific machine learning.

Problem

• Predict structured labelings on Riemannian manifolds
• Combine Laplace-Beltrami framework with assignment flow
• Learn compact parametrization for mutual metric-state dependency

Innovation

• Sigma flow model for structured labeling
• Combines Laplace-Beltrami and assignment flow
• Nonlinear geometric PDE for harmonic mapping