Measure-to-measure interpolation using Transformers

2024-11-07

arXiv.org

AI Summary

This paper investigates the expressive power of Transformers as arbitrary input-to-output measure mappings. Method: We reformulate Transformers from a measure-theoretic perspective, modeling them as differentiable maps on continuous measure spaces—departing from conventional discrete token-based interpretations. Leveraging continuity equations to describe particle dynamics, we design an attention mechanism incorporating spherical geometry constraints and optimal transport theory. Contribution/Results: We propose the first Transformer architecture provably capable of exact matching between arbitrary input–target measure pairs. Under the minimal assumption that a transport map exists between each pair, a single model achieves precise matching for N arbitrary measure pairs. We establish theoretical completeness by proving that Transformers serve as universal interpolators between measures and provide explicit parameter constructions. This work fundamentally characterizes the expressive capacity of Transformers for measure transformation tasks.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Transformers are deep neural network architectures that underpin the recent successes of large language models. Unlike more classical architectures that can be viewed as point-to-point maps, a Transformer acts as a measure-to-measure map implemented as specific interacting particle system on the unit sphere: the input is the empirical measure of tokens in a prompt and its evolution is governed by the continuity equation. In fact, Transformers are not limited to empirical measures and can in principle process any input measure. As the nature of data processed by Transformers is expanding rapidly, it is important to investigate their expressive power as maps from an arbitrary measure to another arbitrary measure. To that end, we provide an explicit choice of parameters that allows a single Transformer to match $N$ arbitrary input measures to $N$ arbitrary target measures, under the minimal assumption that every pair of input-target measures can be matched by some transport map.

Problem

• Investigating Transformers' expressive power as measure-to-measure maps
• Matching arbitrary input measures to arbitrary target measures using Transformers
• Providing explicit parameters to interpolate between different probability measures

Innovation

• Transformers implement measure-to-measure maps
• Use interacting particle systems on sphere
• Match arbitrary input-output measures via transport