Training-Free Universal Approximation by Prompting Random Transformers

2026-08-10

AI Summary

This work investigates whether prompting alone can endow an untrained Transformer with universal function approximation capability. By constructing explicit soft prompts, the authors demonstrate that a single-layer Transformer with random initialization—without any training—can emulate a Nadaraya–Watson kernel regressor and approximate any Hölder continuous function. Leveraging the equivalence between softmax attention and Gaussian kernels, they design prompts by solving a linear system and, through tools from random matrix theory and kernel methods, provide the first approximation-theoretic proof that pretraining is not essential, revealing that prompts alone can encode task-specific information. The analysis yields minimax-optimal approximation rates dependent on the intrinsic dimension of the data manifold and establishes quantitative trade-offs among prompt norm, length, and hidden dimension. Numerical experiments corroborate both the theoretical predictions and the efficacy of the proposed prompt construction.

Abstract

How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in transformer models, and for determining whether task-specific behavior must be stored in model weights or can instead be induced at inference time through the prompt. We show, in an approximation-theoretic sense, that pretraining is optional: a single-layer softmax attention network with random, untrained weights can approximate any Hölder function on a compact manifold when steered by an appropriate soft prompt. Guided by the connection between softmax attention and kernel methods, we construct explicit soft prompts (a prompt per target function, independent of the query) as solutions to linear systems matching attention logits to Gaussian kernel exponents, under which the frozen transformer emulates the classical Nadaraya-Watson kernel estimator. The construction requires only a mild rank condition on the weights, which we show holds almost surely under Gaussian initialization. The prompted network inherits the theoretical guarantees of kernel regression, leading to universal approximation theorems with minimax-optimal rates that depend on the intrinsic dimension. We further quantify the cost of prompting, exposing a tradeoff between the norm of the constructed soft prompt tokens, prompt length, and hidden dimension. Numerical experiments corroborate the constructions and predicted rates.

Problem

• universal approximation
• prompting
• random transformers
• kernel methods
• softmax attention

Innovation

• prompting
• universal approximation
• random transformers
• kernel methods
• Nadaraya-Watson estimator