From Unstructured Communication to Intelligent RAG: Multi-Agent Automation for Supply Chain Knowledge Bases

2025-06-20

AI Summary

Tacit supply-chain knowledge—such as failure-resolution experience—is scattered across unstructured work orders, emails, and chat logs, exhibiting high noise, inconsistency, and poor retrievability, thereby severely limiting RAG system performance. Method: We propose an offline-first LLM-based multi-agent framework featuring three novel, cooperative agents: a category discovery agent, a work-order classification agent, and a knowledge synthesis agent. This framework enables end-to-end automated construction of a structured knowledge base from raw communications, integrating unsupervised clustering, knowledge distillation, and structured generation. Contribution/Results: The resulting knowledge base occupies only 3.4% of the original data volume. When integrated into RAG, it boosts effective answer rate to 48.74% (+10.14 percentage points), reduces invalid responses by 77.4%, and enables fully automated closed-loop resolution for approximately 50% of new work orders.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Multiagent Systems: Teamwork
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Supply chain operations generate vast amounts of operational data; however, critical knowledge such as system usage practices, troubleshooting workflows, and resolution techniques often remains buried within unstructured communications like support tickets, emails, and chat logs. While RAG systems aim to leverage such communications as a knowledge base, their effectiveness is limited by raw data challenges: support tickets are typically noisy, inconsistent, and incomplete, making direct retrieval suboptimal. Unlike existing RAG approaches that focus on runtime optimization, we introduce a novel offline-first methodology that transforms these communications into a structured knowledge base. Our key innovation is a LLMs-based multi-agent system orchestrating three specialized agents: Category Discovery for taxonomy creation, Categorization for ticket grouping, and Knowledge Synthesis for article generation. Applying our methodology to real-world support tickets with resolution notes and comments, our system creates a compact knowledge base - reducing total volume to just 3.4% of original ticket data while improving quality. Experiments demonstrate that our prebuilt knowledge base in RAG systems significantly outperforms traditional RAG implementations (48.74% vs. 38.60% helpful answers) and achieves a 77.4% reduction in unhelpful responses. By automating institutional knowledge capture that typically remains siloed in experts' heads, our solution translates to substantial operational efficiency: reducing support workload, accelerating resolution times, and creating self-improving systems that automatically resolve approximately 50% of future supply chain tickets. Our approach addresses a key gap in knowledge management by transforming transient communications into structured, reusable knowledge through intelligent offline processing rather than latency-inducing runtime architectures.

Problem

• Transforms unstructured supply chain communications into structured knowledge
• Improves RAG systems by preprocessing noisy support ticket data
• Automates knowledge capture to reduce support workload and resolution times

Innovation

• LLMs-based multi-agent system for knowledge structuring
• Offline-first methodology transforms unstructured communications
• Specialized agents for taxonomy, categorization, synthesis