LLM-based Multi-Agent Blackboard System for Information Discovery in Data Science

2025-09-30

AI Summary

To address the challenges of information discovery in large-scale heterogeneous data lakes, insufficient capability of single agents, and rigid dependency on centralized controllers in master–slave multi-agent systems, this paper proposes a large language model (LLM)-driven multi-agent blackboard system. Inspired by the classical blackboard architecture, the system eliminates the need for a predefined central scheduler, enabling agents to autonomously perceive tasks and publish/respond to blackboard messages based on their individual capabilities—thereby achieving decentralized collaborative retrieval. Its core innovation lies in tightly integrating LLM reasoning with a dynamic blackboard mechanism to enhance system flexibility and scalability. Experimental results on multiple benchmarks demonstrate that the proposed approach improves end-to-end task success rate by 13%–57% over both RAG and master–slave multi-agent baselines, while achieving up to a 9% gain in data discovery F1 score.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The rapid advancement of Large Language Models (LLMs) has opened new opportunities in data science, yet their practical deployment is often constrained by the challenge of discovering relevant data within large heterogeneous data lakes. Existing methods struggle with this: single-agent systems are quickly overwhelmed by large, heterogeneous files in the large data lakes, while multi-agent systems designed based on a master-slave paradigm depend on a rigid central controller for task allocation that requires precise knowledge of each sub-agent's capabilities. To address these limitations, we propose a novel multi-agent communication paradigm inspired by the blackboard architecture for traditional AI models. In this framework, a central agent posts requests to a shared blackboard, and autonomous subordinate agents -- either responsible for a partition of the data lake or general information retrieval -- volunteer to respond based on their capabilities. This design improves scalability and flexibility by eliminating the need for a central coordinator to have prior knowledge of all sub-agents' expertise. We evaluate our method on three benchmarks that require explicit data discovery: KramaBench and modified versions of DS-Bench and DA-Code to incorporate data discovery. Experimental results demonstrate that the blackboard architecture substantially outperforms baselines, including RAG and the master-slave multi-agent paradigm, achieving between 13% to 57% relative improvement in end-to-end task success and up to a 9% relative gain in F1 score for data discovery over the best-performing baselines across both proprietary and open-source LLMs. Our findings establish the blackboard paradigm as a scalable and generalizable communication framework for multi-agent systems.

Problem

• Addressing data discovery challenges in large heterogeneous data lakes
• Overcoming limitations of rigid master-slave multi-agent systems
• Improving scalability and flexibility in multi-agent information retrieval

Innovation

• Uses blackboard architecture for multi-agent communication
• Central agent posts requests to shared blackboard
• Autonomous agents volunteer based on capabilities