Ordinary Least Squares as an Attention Mechanism

2025-04-13

AI Summary

A theoretical gap exists between classical statistical modeling—particularly ordinary least squares (OLS) regression—and modern attention mechanisms, hindering principled integration and interpretability. Method: We rigorously reformulate OLS regression as a constrained form of attention, establishing the first exact mathematical equivalence between OLS predictions and attention outputs computed via inner-product similarity. Specifically, we show that OLS is equivalent to a sample-weighted average in embedding space, where weights are determined solely by feature-wise inner products between inputs and queries. Contribution/Results: (1) We provide statisticians with an interpretable, geometrically intuitive lens for understanding attention; (2) we introduce a paradigm shift from explicit coefficient estimation to learned encoding–decoding; and (3) we establish the first rigorous theoretical bridge between classical statistical modeling and Transformer architectures—enabling new interfaces for explainable AI and fostering cross-disciplinary insight.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Computer Vision: Interpretability, Explainability, and Transparency
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

I show that ordinary least squares (OLS) predictions can be rewritten as the output of a restricted attention module, akin to those forming the backbone of large language models. This connection offers an alternative perspective on attention beyond the conventional information retrieval framework, making it more accessible to researchers and analysts with a background in traditional statistics. It falls into place when OLS is framed as a similarity-based method in a transformed regressor space, distinct from the standard view based on partial correlations. In fact, the OLS solution can be recast as the outcome of an alternative problem: minimizing squared prediction errors by optimizing the embedding space in which training and test vectors are compared via inner products. Rather than estimating coefficients directly, we equivalently learn optimal encoding and decoding operations for predictors. From this vantage point, OLS maps naturally onto the query-key-value structure of attention mechanisms. Building on this foundation, I discuss key elements of Transformer-style attention and draw connections to classic ideas from time series econometrics.

Problem

• Connects OLS predictions to attention mechanisms in LLMs
• Reframes OLS as similarity-based in transformed regressor space
• Links Transformer attention to time series econometrics concepts

Innovation

• OLS rewritten as restricted attention module
• Framing OLS as similarity-based method
• Learning optimal encoding and decoding operations