Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

2026-05-13

AI Summary

Existing semantic similarity–based example selection and ordering strategies struggle to effectively support multi-example chain-of-thought in-context learning (CoT-ICL) for reasoning tasks, often yielding unstable performance. This work reveals that CoT-ICL exhibits prompt-setting dependency and order-scaling effects, and proposes treating long contexts as structured curricula, constructing demonstrations according to principles of model interpretability and conceptual progression. To this end, the authors introduce Curvature-based Demonstration Selection (CDS), a method for ranking reasoning examples based on curvature-derived metrics. Experimental results demonstrate that, on geometric reasoning tasks with 64 in-context examples, CDS significantly outperforms existing baselines, achieving performance gains of up to 5.42 percentage points.

Abstract

In-context learning (ICL) adapts large language models (LLMs) to new tasks by conditioning on demonstrations in the prompt without parameter updates. With long-context models, many-shot ICL can use dozens to hundreds of examples and achieve performance comparable to fine-tuning, yet current understanding of its scaling behavior is largely derived from non-reasoning tasks. We study many-shot chain-of-thought in-context learning (CoT-ICL) for reasoning and show that standard many-shot rules do not transfer. Across non-reasoning and reasoning-oriented LLMs and across non-reasoning and reasoning tasks, we find: (i) a setting-dependent scaling effect, where increasing the number of CoT demonstrations is unstable for non-reasoning LLMs and benefits mainly reasoning-oriented LLMs; (ii) similarity-based retrieval helps on non-reasoning tasks but fails on reasoning, since semantic similarity poorly predicts procedural (i.e., CoT) compatibility; and (iii) an order-scaling effect, where performance variance grows with more CoT demonstrations. We interpret these behaviors by viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching, and suggests two principles: (i) demonstrations should be easy for the target model to understand, and (ii) they should be ordered to support a smooth conceptual progression. Guided by the principle, we propose Curvilinear Demonstration Selection (CDS), a simple ordering method that yields up to a 5.42 percentage-point gain on geometry with 64 demonstrations. Overall, our results reframe the long context window from a retrieval buffer into a structured curriculum for in-context test-time learning.

Problem

• in-context learning
• chain-of-thought
• reasoning tasks
• many-shot learning
• demonstration ordering

Innovation

• Chain-of-Thought In-Context Learning
• Many-Shot ICL
• Test-Time Learning
• Demonstration Ordering
• Curvilinear Demonstration Selection