🤖 AI Summary
This paper addresses symbol estimation over noisy wireless channels by proposing a novel “In-Context Estimation” (ICE) modeling framework: the received signal is modeled as a nonlinear noisy observation of transmitted symbols, conditioned on unknown channel context parameters. Unlike linear regression, the optimal ICE estimator is inherently a nonlinear function of the context. Methodologically, we rigorously prove that a single-layer Softmax-attention Transformer, when provided with sufficiently long context prompts, exactly implements the optimal ICE estimator; moreover, its attention weights correspond precisely to the minimizers of the training loss—achieving theoretical–empirical alignment. Our analysis integrates in-context learning (ICL), statistical inference, and asymptotic theory, and is empirically validated using multi-layer Transformers. Experiments across diverse channel models demonstrate that ICE achieves near-oracle estimation performance—matching that attainable with perfect channel knowledge—using only a few received samples, significantly outperforming conventional estimation algorithms.
📝 Abstract
Pre-trained transformers exhibit the capability of adapting to new tasks through in-context learning (ICL), where they efficiently utilize a limited set of prompts without explicit model optimization. The canonical communication problem of estimating transmitted symbols from received observations can be modelled as an in-context learning problem: Received observations are essentially a noisy function of transmitted symbols, and this function can be represented by an unknown parameter whose statistics depend on an (also unknown) latent context. This problem, which we term in-context estimation (ICE), has significantly greater complexity than the extensively studied linear regression problem. The optimal solution to the ICE problem is a non-linear function of the underlying context. In this paper, we prove that, for a subclass of such problems, a single layer softmax attention transformer (SAT) computes the optimal solution of the above estimation problem in the limit of large prompt length. We also prove that the optimal configuration of such transformer is indeed the minimizer of the corresponding training loss. Further, we empirically demonstrate the proficiency of multi-layer transformers in efficiently solving broader in-context estimation problems. Through extensive simulations, we show that solving ICE problems using transformers significantly outperforms standard approaches. Moreover, just with a few context examples, it achieves the same performance as an estimator with perfect knowledge of the latent context.