Transformers perform adaptive partial pooling

2026-02-03

AI Summary

This study investigates the generalization mechanisms of language models in low-frequency yet non-novel contexts, proposing that they may exhibit adaptive partial pooling akin to hierarchical regression models. Focusing on next-token prediction with GPT-2, the work leverages a hierarchical regression framework to quantify how the model’s reliance on information from similar contexts dynamically adjusts during training as a function of context frequency, type count, and variability. The research reveals, for the first time, that Transformers display adaptive partial pooling behavior consistent with classical statistical models, demonstrating their capacity to modulate cross-context information sharing in a principled, experience-driven manner. These findings bridge the generalization mechanisms of deep learning with those of traditional statistical modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Because language is creative, any reasonable language model must generalize, deciding what to say in novel contexts by using information from similar contexts. But what about contexts that are not novel but merely infrequent? In hierarchical regression, the model's predictions for behavior in a context are affected by observations from other similar contexts to the extent that 1) the current context is infrequent and 2) different contexts behave similarly. This is called adaptive partial pooling of evidence. This paper shows that next-word predictions of a transformer (GPT2) are increasingly unaffected by observations from outside the current context across epochs of training (the amount of pooling reduces with training), and that the extent of pooling is affected by context frequency, context number (type frequency) and context variability in a similar way to hierarchical regression. These characteristics of learning in transformers are argued to be realistic on both rational and empirical grounds.

Problem

• adaptive partial pooling
• transformers
• context frequency
• hierarchical regression
• language models

Innovation

• adaptive partial pooling
• transformer
• hierarchical regression
• context frequency
• generalization