Can In-Context Learning Support Intrinsic Curiosity?

2026-06-17

AI Summary

This work addresses the challenge of scalable, intrinsic curiosity–driven data collection without relying on costly gradient updates. It proposes leveraging the in-context learning (ICL) capability of sequence models to construct a parameter-update-free world model, wherein prediction errors and counterfactual context manipulations drive a learning-progress–based exploration strategy. Theoretically, the study establishes—for the first time—that ICL can provide an unbiased estimate of learning progress in non-sequential tasks, while also revealing an inherent bias in general Markov decision processes. Empirically, the method effectively learns near-optimal data-gathering behaviors in both continuous and symbolic environments and achieves asymptotic convergence of intrinsic rewards in non-sequential settings.

Abstract

Effective machine learning depends not only on how we model data, but also on what data we choose to collect. While large sequence models have revolutionized data modeling, the problem of automated data selection, or "intrinsic curiosity", remains a significant challenge. Classic approaches incentivize exploration by rewarding an agent based on its "learning progress", which measures how much a newly acquired observation improves a world model's predictive ability. However, evaluating these rewards traditionally requires expensive inner loops of gradient descent updates within each trajectory, rendering them computationally impractical at scale. In this work, we investigate whether the emergent in-context learning (ICL) capabilities of sequence models can eliminate this bottleneck by serving as immediate, update-free world models. Specifically, we evaluate whether an exploration policy can be trained to maximize learning progress, using solely the prediction errors and counterfactual context manipulations of an in-context learner. We first prove that in general Markov decision processes, this is in fact impossible in an unbiased way: the resulting intrinsic rewards either suffer from nuisance terms that bias their estimation of true learning progress, or they cannot be implemented using an in-context learner's prediction errors. Conversely, we prove a positive result for a broad subclass of non-temporal settings, encompassing active learning and Bayesian Experimental Design: here, ICL-derived rewards successfully bound and asymptotically converge to the true learning progress. We corroborate our theory with controlled experiments across continuous and symbolic environments, demonstrating that our ICL-driven framework successfully trains curious data-collection policies that explore optimally.

Problem

• intrinsic curiosity
• in-context learning
• learning progress
• automated data selection
• world models

Innovation

• in-context learning
• intrinsic curiosity
• learning progress
• active learning
• Bayesian experimental design