Offline Preference-Based Trajectory Evaluation

2026-06-16

AI Summary

Traditional offline evaluation relies solely on task success or failure, ignoring intermediate progress and resulting in identical assessments for a large proportion of samples, which undermines statistical efficiency and system discriminability. This work introduces trajectory-level temporal preferences into offline evaluation for the first time, proposing a preference-based trajectory evaluation method that directly compares agent trajectories through progress-aware preferences and return distributions over time. The approach establishes an evaluation framework integrating preference modeling, pairwise trajectory comparison, and joint temporal-progress analysis. Evaluated across diverse agents and interactive benchmarks, it reduces evaluation tie rates from approximately 75% to 35%, substantially improving discriminative power, ranking stability, and data efficiency. Furthermore, the study reveals that benchmark saturation may stem not from agent capabilities but from limitations inherent in conventional evaluation metrics themselves.

Abstract

Offline evaluation of agentic systems often collapses trajectories to terminal success, discarding information about partial progress and inducing widespread ties, creating substantial statistical inefficiency by reducing effective sample size and weakening the ability to distinguish systems. We propose preference-based trajectory evaluation, which compares trajectories directly through temporal preferences over progress and time-to-return profiles. We find that, across diverse agentic and interactive benchmarks, standard success-based metrics produce tied comparisons on roughly 75% of instances, whereas trajectory-aware preferences reduce ties to roughly 35%, improving discriminative power, ranking stability, and data efficiency. Our results suggest that benchmark saturation, often attributed to poor data collection or problem difficulty, may also be explained by the choice of evaluation measure.

Problem

• offline evaluation
• trajectory evaluation
• preference-based comparison
• statistical inefficiency
• benchmark saturation

Innovation

• preference-based evaluation
• trajectory comparison
• offline evaluation
• agentic systems
• statistical efficiency