DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agents

2026-09-13

AI Summary

为解决长任务执行中评估方法的局限性，DynSTEER通过动态阶段式轨迹评估和容错里程碑图来提高评估准确性并减少资源浪费。

Abstract

Large language model agents are increasingly deployed for long-horizon task execution. However, current evaluation paradigms face three major limitations: terminal-only assessment ignores intermediate processes and struggles to localize errors efficiently and accurately, single-reference matching penalizes valid alternative solution paths, and post-hoc trajectory judging incurs high costs without the ability to halt failed runs early. To address these issues, we propose DynSTEER, a dynamic stage-wise trajectory evaluation framework for agents. DynSTEER segments rollouts into stages anchored by key completed actions, focusing evaluation on essential milestones with adequate context while enabling targeted strategy adjustments. It compiles a path-tolerant milestone graph from public task views to respect diverse legitimate strategies without leaking ground truth. Furthermore, it adaptively routes evaluation queries across multi-tier judges and halts unrecoverable executions online to curb resource waste. Experiments demonstrate that DynSTEER improves evaluation discriminability across LLM agents by 85.2\% over native evaluation, separates all model pairs with statistical significance, and saves 34.51\% of execution steps on failed rollouts.

Problem

• terminal-only assessment
• single-reference matching
• post-hoc trajectory judging

Innovation

• dynamic stage-wise evaluation
• path-tolerant milestone graph
• adaptive query routing
• execution halting