Blue Data Intelligence Layer: Streaming Data and Agents for Multi-source Multi-modal Data-Centric Applications

2026-04-16

AI Summary

Existing NL2SQL systems struggle to handle iterative queries in real-world scenarios that involve multiple heterogeneous data sources, multimodal inputs, and external commonsense knowledge. This work proposes Blue, a Data Intelligence Layer (DIL) that treats large language models, web resources, and user interactions as a novel form of database. Blue employs a data registry to uniformly manage metadata from diverse sources and introduces a data planner that generates declarative query plans integrating relational operators with multimodal operators. This approach dynamically orchestrates cross-source retrieval, multimodal reasoning, and result synthesis, effectively overcoming the limitations of traditional single-database NL2SQL systems in two representative interactive settings.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

NL2SQL systems aim to address the growing need for natural language interaction with data. However, real-world information rarely maps to a single SQL query because (1) users express queries iteratively (2) questions often span multiple data sources beyond the closed-world assumption of a single database, and (3) queries frequently rely on commonsense or external knowledge. Consequently, satisfying realistic data needs require integrating heterogeneous sources, modalities, and contextual data. In this paper, we present Blue's Data Intelligence Layer (DIL) designed to support multi-source, multi-modal, and data-centric applications. Blue is a compound AI system that orchestrates agents and data for enterprise settings. DIL serves as the data intelligence layer for agentic data processing, to bridge the semantic gap between user intent and available information by unifying structured enterprise data, world knowledge accessible through LLMs, and personal context obtained through interaction. At the core of DIL is a data registry that stores metadata for diverse data sources and modalities to enable both native and natural language queries. DIL treats LLMs, the Web, and the User as source 'databases', each with their own query interface, elevating them to first-class data sources. DIL relies on data planners to transform user queries into executable query plans. These plans are declarative abstractions that unify relational operators with other operators spanning multiple modalities. DIL planners support decomposition of complex requests into subqueries, retrieval from diverse sources, and finally reasoning and integration to produce final results. We demonstrate DIL through two interactive scenarios in which user queries dynamically trigger multi-source retrieval, cross-modal reasoning, and result synthesis, illustrating how compound AI systems can move beyond single database NL2SQL.

Problem

• NL2SQL
• multi-source
• multi-modal
• data integration
• semantic gap

Innovation

• Data Intelligence Layer
• Multi-source Multi-modal Data
• Compound AI System
• NL2SQL
• Data Planner