Agentic Context Learning with Self-Discovered Specification

2026-07-09

AI Summary

Large language models struggle in in-context learning to extract implicit local specifications—such as formatting rules, structural constraints, and completeness requirements—from complex examples for unseen tasks, which limits their performance. This work identifies specification acquisition as a critical bottleneck through empirical analysis and introduces PSCI, a method that leverages large models to automatically distill local specifications and integrates adversarial validation and repair mechanisms to strengthen reasoning. PSCI is the first approach to systematically demonstrate the pivotal role of specification acquisition, achieving substantial performance gains with minimal intervention. On CL-Bench, PSCI improves the success rate of GPT-5.1 by 5.59 percentage points, reaching 28.14%, and consistently replicates these gains on Qwen3.5-27B and Gemini 3 Pro.

Abstract

Context learning is an emerging inference-time task where LLMs must learn and apply novel, task-specific knowledge from intricate contexts absent from pre-training; even frontier models score under 24% task success. In this work, we conduct a comprehensive empirical study to understand why this setting remains difficult. A natural hypothesis is that failures stem from content access; yet across twelve retrieval, reflection, and verification baselines on CL-Bench, an extensive context learning benchmark, we find limited gains over direct full-context prompting. Further failure analysis reveals a key finding: unlike typical long-context tasks such as long document understanding, context learning requires not only recovering local content but also acquiring local specifications that are often unspecified in the query but distributed across the context: domain-specific formats, local rules, and completeness conditions. Across all 31,592 rubric items, we find that 55.4% clearly evaluate specification acquisition, while only 22.6% evaluate content acquisition. Moreover, despite 76.7% of specifications being unspecified in the user query, 95.5% are traceable to the context, indicating these are learnable obligations rather than hidden requirements. To validate this diagnosis, we design a deliberately simple intervention PSCI (private specification-contract induction) which extracts local specifications and enforces them through adversarial checking and repair; PSCI achieves state-of-the-art 28.14% with GPT-5.1 (+5.59 pp absolute and +24.8% relative) on CL-Bench, replicated on Qwen3.5-27B (+5.28 pp) and Gemini 3 Pro (+6.17 pp). Seventeen ablations further isolate the role of task-specific specifications. Overall, our results suggest context learning hinges on not only content acquisition but also specification acquisition.

Problem

• context learning
• specification acquisition
• large language models
• task-specific knowledge
• local specifications

Innovation

• context learning
• specification acquisition
• PSCI
• local specifications
• CL-Bench