Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens

2024-10-24

arXiv.org

AI Summary

Modern large language models rely on high-dimensional token representations over long sequences, yet lack simple, analytically tractable benchmark models for sequence modeling. Method: We propose Bilinear Sequence Regression (BSR), the first exactly solvable token-sequence modeling paradigm that incorporates skip connections and captures long-range dependencies observed in real-world large models. Leveraging statistical physics—specifically mean-field theory—and high-dimensional asymptotic analysis, we derive the Bayes-optimal generalization error in closed form. Contribution/Results: We characterize the nontrivial dynamics of gradient descent on BSR, design a message-passing algorithm achieving the theoretical performance limit, and empirically demonstrate that BSR substantially outperforms vectorized linear regression baselines. Our framework establishes an analytically accessible, statistically grounded benchmark—rooted in statistical physics—for rigorously studying sequence modeling in large language models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Current progress in artificial intelligence is centered around so-called large language models that consist of neural networks processing long sequences of high-dimensional vectors called tokens. Statistical physics provides powerful tools to study the functioning of learning with neural networks and has played a recognized role in the development of modern machine learning. The statistical physics approach relies on simplified and analytically tractable models of data. However, simple tractable models for long sequences of high-dimensional tokens are largely underexplored. Inspired by the crucial role models such as the single-layer teacher-student perceptron (aka generalized linear regression) played in the theory of fully connected neural networks, in this paper, we introduce and study the bilinear sequence regression (BSR) as one of the most basic models for sequences of tokens. We note that modern architectures naturally subsume the BSR model due to the skip connections. Building on recent methodological progress, we compute the Bayes-optimal generalization error for the model in the limit of long sequences of high-dimensional tokens, and provide a message-passing algorithm that matches this performance. We quantify the improvement that optimal learning brings with respect to vectorizing the sequence of tokens and learning via simple linear regression. We also unveil surprising properties of the gradient descent algorithms in the BSR model.

Problem

• Develops bilinear sequence regression for long high-dimensional token sequences
• Computes Bayes-optimal generalization error for token sequence learning
• Analyzes gradient descent properties in bilinear sequence regression

Innovation

• Introduces bilinear sequence regression for token sequences
• Computes Bayes-optimal generalization error analytically
• Proposes message-passing algorithm matching optimal performance