Dual Filter: A Mathematical Framework for Inference using Transformer-like Architectures

2025-05-01

AI Summary

This work addresses causal nonlinear prediction for observation sequences generated by hidden Markov models (HMMs), aiming to derive a Transformer-like mathematical architecture from first principles—not by modeling existing Transformers. Methodologically, it reformulates minimum mean-square error prediction as an optimal control problem on the space of probability measures, derives a fixed-point equation, and introduces—novelty—the “dual filter” iterative algorithm, which rigorously yields an analytical form identical to the decoder-only Transformer architecture. Theoretically, it establishes a deep connection among prediction, stochastic optimal control, and probabilistic transport dynamics, revealing the attention mechanism as the evolution of probability measures under optimal transport. Empirically, the derived architecture achieves prediction performance comparable to standard Transformers under identical parameter configurations.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This paper presents a mathematical framework for causal nonlinear prediction in settings where observations are generated from an underlying hidden Markov model (HMM). Both the problem formulation and the proposed solution are motivated by the decoder-only transformer architecture, in which a finite sequence of observations (tokens) is mapped to the conditional probability of the next token. Our objective is not to construct a mathematical model of a transformer. Rather, our interest lies in deriving, from first principles, transformer-like architectures that solve the prediction problem for which the transformer is designed. The proposed framework is based on an original optimal control approach, where the prediction objective (MMSE) is reformulated as an optimal control problem. An analysis of the optimal control problem is presented leading to a fixed-point equation on the space of probability measures. To solve the fixed-point equation, we introduce the dual filter, an iterative algorithm that closely parallels the architecture of decoder-only transformers. These parallels are discussed in detail along with the relationship to prior work on mathematical modeling of transformers as transport on the space of probability measures. Numerical experiments are provided to illustrate the performance of the algorithm using parameter values used in researchscale transformer models.

Problem

• Develops a framework for causal prediction using HMM observations
• Derives transformer-like architectures from first principles
• Proposes dual filter algorithm for solving optimal control problem

Innovation

• Mathematical framework for causal nonlinear prediction
• Optimal control approach reformulates prediction objective
• Dual filter algorithm parallels transformer architecture