SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

2026-06-16

AI Summary

Existing evaluation methods struggle to comprehensively assess whether self-evolving large language model (LLM) agents genuinely acquire reusable improvements when updating their execution frameworks or instead suffer from overfitting, capability degradation, and increased costs. To address this, this work proposes SEAGym—the first multidimensional dynamic evaluation framework tailored for self-evolving LLM agents. SEAGym integrates a dynamic task source (based on the Harbor-compatible benchmark), batch training, frozen validation, replay diagnostics, snapshot preservation, and cost tracking, enabling both in- and out-of-distribution transfer analysis and behavioral pattern dissection. Experiments on Terminal-Bench 2.0 and HLE reveal stark differences among update strategies in terms of retention, generalization, and stability, demonstrating that multi-view evaluation effectively uncovers complex degradation or improvement phenomena invisible to single metrics.

Abstract

Self-evolving LLM-based agents improve mainly by changing their agent harness: the structured execution layer around a base model, including prompts, memory, tools, middleware, runtime state, and the model-tool interaction loop. Existing evaluations often reduce this process to isolated task scores or a single sequential curve, obscuring whether an update produces reusable improvement, overfits recent tasks, increases cost, or harms older behavior. We introduce SEAGym, an evaluation environment for measuring agent harness updates across training, validation, test, replay, and cost records. SEAGym turns Harbor-compatible benchmarks into dynamic self-evolution task sources with train batches, frozen update-validation, held-out ID and OOD transfer views, replay diagnostics, and saved snapshot and metric records. Instantiating SEAGym on Terminal-Bench 2.0 and HLE, we compare ACE, TF-GRPO, and AHE under a shared epoch/batch protocol. The results show that these evaluation views provide complementary signals about the evolution process: frequent updates may fail to improve held-out performance, useful intermediate snapshots may collapse later, and source diversity and model backend can affect harness reliability.

Problem

• self-evolving agents
• LLM agents
• agent harness
• evaluation environment
• harness updates

Innovation

• self-evolving agents
• agent harness
• evaluation framework
• LLM-based agents
• replay diagnostics