What Could the Agent See at 19:05? Generating Temporal Enterprise Scenarios from Real Research and Replaying Them to Evaluate Agents

2026-08-02

AI Summary

Existing offline evaluation methods rely on static snapshots, which fail to capture the dynamic evolution of multi-application data states and permission visibility in enterprise environments over time, leading to distorted assessments. This work proposes the first time-sensitive, multi-application-aware framework for efficient reconstruction and replay in enterprise settings. By leveraging schema inference and deterministic precomputation of critical historical states, the framework compactly stores these states using a delta-caching mechanism, thereby avoiding redundant full-tenant reconstructions and preventing future information leakage. Integrated with large language models, the system enables reproducible replay from any point in time. This architecture substantially enhances the accuracy, efficiency, and reproducibility of evaluating enterprise AI agents in realistic, dynamically evolving scenarios.

Abstract

Enterprise AI agents act across many apps whose data changes continuously, so an answer is correct only relative to what data existed and who could see it at the moment it was asked. Offline evaluation today grades against a single static snapshot, effectively the end of the episode. So, it can only evaluate one situation, the final one, even though every earlier moment of the episode is a different situation that invites its own realistic questions with its own correct answers. Recreating each of those moments as a separate snapshot would mean re-provisioning a whole tenant per instant, which is prohibitively costly; and even a single snapshot leaks future state hidden inside records and cannot represent the multi-app, time-ordered way real work happens. Our system closes two gaps at once: it generates a realistic, persona-driven, temporally-evolving enterprise world from real research, and replays that world at any chosen moment to evaluate any pluggable agent. A schema-inferred temporal description drives a deterministic-plus-LLM rebuild of each record's past state; because the queryable moments are finite, all rebuilds are precomputed into a compact difference cache, making evaluation a fast, reproducible lookup with no model in the path. We describe the design, an architecture spanning both flows, and early experience evaluating enterprise agents.

Problem

• Enterprise AI agents
• temporal evaluation
• offline evaluation
• time-evolving scenarios
• data visibility

Innovation

• temporal enterprise scenarios
• agent evaluation
• difference cache
• schema-inferred reconstruction
• persona-driven simulation