🤖 AI Summary
Traditional vector retrieval struggles to handle complex queries requiring structured reasoning in industrial knowledge graphs. This work constructs an aerospace supply chain knowledge graph comprising 46 node types and 64 relation types, and introduces the “operator vocabulary” hypothesis, positing that the bottleneck in graph reasoning lies not in model intelligence but in the availability of computational primitives. Guided by this insight, the authors design an LLM-driven query planner that integrates nine graph traversal primitives and six graph computation tools, forming a structure-aware retrieval-augmented generation framework. Evaluated on 23 queries spanning ten intent categories, the approach achieves an F1 score of 0.632, significantly outperforming a customized processor (0.472), and further exposes a systematic bias in existing entity-level F1 metrics when assessing structured queries.
📝 Abstract
Retrieval-Augmented Generation (RAG) fails systematically on queries requiring structural reasoning over interconnected entities. We compare eight retrieval architectures for aerospace supply chain intelligence, progressing from text retrieval through graph traversal to graph computation. Using a 46-node knowledge graph with 64 typed edges, we evaluate 23 queries across 10 intent categories and demonstrate that five query classes are structurally unreachable for vector retrieval. Our central finding is the operator vocabulary thesis: the barrier to LLM-based graph reasoning is not model intelligence but the computational operators available as tools. An LLM Query Planner with 9 typed traversal primitives outperforms bespoke handlers (F1 = 0.632 vs. 0.472) while generalizing to unseen queries. Adding 6 graph computation tools, the LLM selectively adopts them for exactly the query categories where traversal fails. We also identify a measurement gap: entity-level F1 systematically underscores structural queries where comprehensive answers are correct.