Fine-Tuning Dynamics of In-Context Factual Recall in Transformers

2026-05-26

AI Summary

This work investigates how large language models leverage parametric factual knowledge during in-context learning and formally introduces the task of in-context fact recall (IC-recall). By constructing MLP-based associative memories grounded in (subject, relation, answer) triples, the authors theoretically demonstrate that a single-layer Transformer can achieve efficient recall with only a polylogarithmic number of samples relative to the number of triples, by converging to specific pairwise attention patterns. Experimental results confirm that even when the MLP layers are not explicitly engineered but solely pretrained, the model rapidly acquires this attention mechanism and accurately retrieves factual knowledge, showing strong alignment between theory and empirical findings.

Abstract

In-context learning \ -- performing tasks based on examples given in the prompt \ -- is an important capability that has emerged in large language models and has received significant attention in both theory and practice. Existing theoretical work often focuses on settings where the learning uses information purely from the prompt. However, many practical instances of in-context learning require the model to retrieve factual knowledge stored in the model's parameters, with the context serving to identify which knowledge is relevant. In this work, we study how in-context learning leverages factual knowledge recall. We formalize this behavior by introducing the \emph{in-context factual recall (IC-recall)} task, where a transformer is provided a context of (subject, answer) pairs generated from a hidden relation, along with a query subject, and must both infer this hidden relation and retrieve the corresponding answer. Factual knowledge is modeled by the transformer having access to a simple pre-constructed MLP associative memory storing (subject, relation, answer) triplets. We analyze the supervised fine-tuning dynamics of a one-layer transformer on IC-recall data and prove that the model successfully performs IC-recall by converging to a particular pairwise attention pattern. This fine-tuning stage requires a very small number of samples \ -- only polylogarithmic in the number of stored knowledge triplets. Experiments verify our theoretical predictions and show that the pairwise attention pattern emerges even when the MLP layer is pretrained instead of constructed.

Problem

• in-context learning
• factual recall
• transformers
• associative memory
• fine-tuning dynamics

Innovation

• in-context learning
• factual recall
• transformer dynamics
• associative memory
• attention pattern