SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment

2026-04-10

AI Summary

Current large language model agents are constrained by static toolsets and inter-task amnesia, hindering cross-task experience accumulation and strategic optimization. This work proposes a formal digital embodiment definition for Self-Evolving Agents (SEA) and introduces SEA-Eval, the first benchmark tailored for evaluating SEA capabilities. Designed around sequential task streams, SEA-Eval jointly assesses in-task execution reliability and long-term evolutionary performance. The framework incorporates quantitative metrics for cross-task continuous evolution, overcoming the limitations of traditional episodic evaluations. Experimental results reveal a significant evolutionary bottleneck in state-of-the-art agents: under equivalent success rates, token consumption varies by up to 31.2×, and temporal analysis shows clearly divergent evolutionary trajectories.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Evolutionary Learning
• Search and Optimization: Evolutionary Computation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Current LLM-based agents demonstrate strong performance in episodic task execution but remain constrained by static toolsets and episodic amnesia, failing to accumulate experience or optimize strategies across task boundaries. While the Self-Evolving Agent (SEA) paradigm has been previously proposed, this paper contributes a new formal definition of SEA grounded in digital embodiment and continuous cross-task evolution, and introduces SEA-Eval, the first benchmark designed to evaluate SEA characteristics across two dimensions, intra-task execution reliability and long-term evolutionary performance. By organizing tasks into sequential streams and analyzing Success Rate and Token Consumption over time, SEA-Eval quantifies evolutionary gain and structural stability in ways that existing episodic benchmarks cannot. Empirical evaluations reveal a significant evolutionary bottleneck in current state-of-the-art frameworks, where identical success rates mask up to 31.2 times differences in token consumption and divergent evolutionary trajectories under sequential analysis. SEA-Eval provides a rigorous scientific foundation for advancing agents from mere task executors toward genuinely self-evolving digital entities.

Problem

• Self-Evolving Agents
• Episodic Assessment
• Cross-task Evolution
• Agent Evaluation
• Digital Embodiment

Innovation

• Self-Evolving Agent
• SEA-Eval
• continuous cross-task evolution
• digital embodiment
• evolutionary bottleneck