EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

2026-07-06

AI Summary

Current agent evaluation frameworks struggle to assess self-evolution and cross-task transfer capabilities grounded in reusable procedural experience—such as searching, debugging, and verification. To address this gap, this work introduces EvoAgentBench, the first fine-grained benchmark specifically designed for evaluating procedural capability transfer. By parsing execution traces, the framework extracts and normalizes discrete procedural “capability” units, constructs capability graphs across four domains, and defines a training/test task split (528/267) to rigorously measure capability reuse. Experiments demonstrate that human-curated capabilities can be reliably transferred across different model families, whereas existing automated methods fail to consistently improve performance across all settings.

Abstract

Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification. Yet current evaluations do not isolate this form of transfer. Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse. We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work. EvoAgentBench extracts trace-grounded Abilities from agent executions, canonicalizes them into operational units, and builds domain-specific Ability Graphs linking tasks that share procedural overlap. By design, every test task is backed by verified training-side Ability support. Across a 528/267 train/test split, two scaffolds, and three backbones, curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. EvoAgentBench shifts self-evolution evaluation from aggregate accuracy comparison to fine-grained diagnosis of experience encoding, routing, and uptake. The benchmark is publicly available at https://huggingface.co/datasets/EverMind-AI/EvoAgentBench.

Problem

• agent self-evolution
• ability transfer
• procedural reuse
• benchmarking
• long-horizon LLM systems

Innovation

• agent self-evolution
• ability transfer
• procedural reuse
• ability graph
• trace-grounded abilities