In-Context Function Learning in Large Language Models

2026-02-12

AI Summary

This work investigates how large language models (LLMs) perform function inference from few-shot examples in in-context learning and establishes, for the first time, a systematic connection between this capability and Gaussian process (GP) theory. Through controlled experiments, LLMs are presented with samples drawn from multivariate scalar functions governed by known GP priors, and their prediction errors are evaluated as a function of the number of observed examples, benchmarked against empirical GP regression and 1-nearest-neighbor baselines. The study introduces a framework to quantify LLMs’ functional learning ability, revealing that their learning curves are shaped by an implicit kernel that converges toward the GP lower bound with increasing data. Initially biased toward less smooth kernels, LLMs can be post-trained to adapt to smoother kernels, substantially improving sample efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) can learn from a few demonstrations provided at inference time. We study this in-context learning phenomenon through the lens of Gaussian Processes (GPs). We build controlled experiments where models observe sequences of multivariate scalar-valued function samples drawn from known GP priors. We evaluate prediction error in relation to the number of demonstrations and compare against two principled references: (i) an empirical GP-regression learner that gives a lower bound on achievable error, and (ii) the expected error of a 1-nearest-neighbor (1-NN) rule, which gives a data-driven upper bound. Across model sizes, we find that LLM learning curves are strongly influenced by the function-generating kernels and approach the GP lower bound as the number of demonstrations increases. We then study the inductive biases of these models using a likelihood-based analysis. We find that LLM predictions are most likely under less smooth GP kernels. Finally, we explore whether post-training can shift these inductive biases and improve sample-efficiency on functions sampled from GPs with smoother kernels. We find that both reinforcement learning and supervised fine-tuning can effectively shift inductive biases in the direction of the training data. Together, our framework quantifies the extent to which LLMs behave like GP learners and provides tools for steering their inductive biases for continuous function learning tasks.

Problem

• in-context learning
• large language models
• Gaussian Processes
• inductive biases
• function learning

Innovation

• in-context learning
• Gaussian Processes
• inductive biases
• function learning
• post-training alignment