Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents

2026-08-15

AI Summary

This study addresses the challenge of one-shot learning for reusable agent skills in noisy environments by proposing the Online Harness Learning framework. Its core innovation lies in a novel "context-to-skill compilation" mechanism that distills single-execution experiences into structured skills, enabling cross-domain adaptive evolution through frozen-model co-optimization. This approach effectively overcomes bottlenecks in topic-level adaptation and on-the-fly learning. Extensive validation across five real-world benchmarks demonstrates that the framework significantly enhances both one-shot skill compilation efficiency and generalization capabilities. Consequently, this work establishes a new paradigm for continuous agent learning in open-ended environments, offering a robust solution for acquiring transferable skills from limited, noisy interactions without extensive retraining.

Abstract

Learning from experience is critical for developing capable, self-improving large language model (LLM) agents. Existing methods typically extract knowledge from accumulated trajectories via reflection, memory, rules, or skills. However, agents in realistic environments continuously encounter novel tasks, often offering only a one-shot opportunity to improve. These executions yield rich but highly noisy contexts, entangling broadly useful lessons with task-specific artifacts. Critically, prior works rarely validate their effectiveness on complex real-world tasks or isolate the underlying drivers of improvement. To address these gaps, we formulate online harness learning, where a frozen agent improves by continually updating a structured harness across sequential tasks. This formulation enables a systematic study of key self-improvement factors through our proposed Evo-Harness. At its core, context-to-harness skill compilation distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation. To demonstrate the efficacy of one-shot skill compilation, we evaluate across five realistic benchmarks (TerminalBench2, SWE-bench, CL-Bench, -bench, WebArena-Infinity). Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly. Our code is available at https://github.com/A-EVO-Lab/a-evolve/tree/release/evo-harness.

Problem

• Self-Evolving Agents
• One-Shot Learning
• Noisy Context
• Online Harness Learning
• Skill Compilation

Innovation

• Context-to-Harness Skill Compilation
• Online Harness Learning
• Self-Evolving Agents
• One-Shot Learning
• Structured Skill Harnesses