Discovery Agents for Real-Time Analytics: Toward Proactive Insight Systems

2026-05-26

AI Summary

This work addresses the limitations of traditional real-time analytics systems, which rely on manually defined queries and struggle to proactively uncover the vast array of potential insights within complex, dynamic data streams. To overcome this, the authors propose a multi-agent architecture that establishes a continuous closed-loop process for autonomous insight discovery, encompassing hypothesis generation, compilation of executable analyses, result validation, and visualization. A key innovation is the introduction of a contract-driven design based on typed intermediate artifacts, which ensures modularity, observability, lineage tracking, and secure execution of dynamic analyses. The system leverages Kafka as its event coordination backbone and Flink for stream processing, integrating large language models to power specialized agents. Empirical evaluations in retail, financial, and public data scenarios demonstrate an effective paradigm shift from query-driven to proactive discovery-driven analytics.

Abstract

Modern analytics systems are fundamentally reactive, requiring users to define queries over increasingly complex and continuously evolving data. In real-time streaming environments, this paradigm breaks down, as the space of potential insights becomes too large to enumerate manually. We present a multi-agent architecture for autonomous insight discovery over real-time data streams. The system implements a continuous discovery loop in which agents generate hypotheses, compile them into executable analytics, validate generated artifacts, and produce visualizations and deployable applications. The architecture leverages Apache Kafka for event-driven coordination, Apache Flink for stream processing, and large language models to implement specialized agents. A key contribution is a contract-driven design based on typed intermediate artifacts, enabling modularity, observability, lineage, and safer execution of dynamically generated analytics. Through use cases in retail, finance, and public data, we show how this architecture supports a shift from query-driven analytics to proactive, discovery-driven systems.

Problem

• real-time analytics
• proactive insight
• data streams
• autonomous discovery
• reactive querying

Innovation

• multi-agent architecture
• real-time analytics
• contract-driven design
• large language models
• stream processing