Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel

2026-03-12

AI Summary

Existing conversational agents for time-series data analysis suffer from limited state awareness, inadequate support for event-specific queries, and poor domain adaptability, compounded by a lack of expressive and customizable evaluation methodologies. To address these gaps, this work proposes AgentFuel—an end-to-end evaluation generation framework that systematically constructs high-fidelity, context-aware benchmarks by integrating domain knowledge injection, synthetic data generation, and query template design. AgentFuel is the first to bridge the expressiveness and customization gap in evaluating time-series agents, and has been integrated into an open-source platform. Empirical studies reveal critical shortcomings in mainstream frameworks and demonstrate that AgentFuel effectively guides agent performance improvements—such as in GEPA—with its generated benchmarks publicly released.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent Communication
• Machine Learning: Time-Series/Data Streams

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Across many domains (e.g., IoT, observability, telecommunications, cybersecurity), there is an emerging adoption of conversational data analysis agents that enable users to "talk to your data" to extract insights. Such data analysis agents operate on timeseries data models; e.g., measurements from sensors or events monitoring user clicks and actions in product analytics. We evaluate 6 popular data analysis agents (both open-source and proprietary) on domain-specific data and query types, and find that they fail on stateful and incident-specific queries. We observe two key expressivity gaps in existing evals: domain-customized datasets and domain-specific query types. To enable practitioners in such domains to generate customized and expressive evals for such timeseries data agents, we present AgentFuel. AgentFuel helps domain experts quickly create customized evals to perform end-to-end functional tests. We show that AgentFuel's benchmarks expose key directions for improvement in existing data agent frameworks. We also present anecdotal evidence that using AgentFuel can improve agent performance (e.g., with GEPA). AgentFuel benchmarks are available at https://huggingface.co/datasets/RockfishData/TimeSeriesAgentEvals.

Problem

• timeseries data analysis agents
• expressive evals
• domain-customized datasets
• stateful queries
• incident-specific queries

Innovation

• AgentFuel
• timeseries data agents
• customizable evals
• expressive benchmarks
• conversational data analysis