A Survey of In-Context Reinforcement Learning

2025-02-11

AI Summary

This paper systematically investigates in-context reinforcement learning (ICRL)—a paradigm enabling agents to adapt to novel tasks solely through contextual cues (e.g., action-observation histories) without parameter updates. We formally define ICRL, rigorously distinguish it from meta-RL and prompt tuning, and propose a unified taxonomy covering architecture design (e.g., Transformer-based backbones), implicit policy modeling, context encoding mechanisms, and generalization evaluation protocols. Synthesizing over one hundred studies, we identify ICRL’s promise in few-shot transfer, online adaptation, and neuro-symbolic integration. Yet we pinpoint two fundamental challenges: limited scalability and insufficient theoretical foundations. Our work establishes the first structured conceptual framework for ICRL, clarifies its methodological boundaries, and provides a roadmap for future research—including scalable context representations, formal generalization guarantees, and principled integration of symbolic reasoning.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement learning (RL) agents typically optimize their policies by performing expensive backward passes to update their network parameters. However, some agents can solve new tasks without updating any parameters by simply conditioning on additional context such as their action-observation histories. This paper surveys work on such behavior, known as in-context reinforcement learning.

Problem

• Optimizing RL policies without parameter updates
• Exploring in-context reinforcement learning methods
• Surveying RL agents' context-based task solving

Innovation

• In-context learning without parameter updates
• Leveraging action-observation histories
• Surveying in-context reinforcement learning techniques