Indexing the Unreadable: LLM-Native Recursive Construction and Search of Service Taxonomies

2026-05-27

AI Summary

In the era of Internet of Agents (IoA), large language models (LLMs) struggle to efficiently and accurately retrieve from vast pools of callable services due to limited context windows and intermediate information loss. To address this challenge, this work proposes A2X—a progressive service disclosure mechanism tailored for LLMs—that pioneers a native, automated framework for hierarchical service categorization and recursive retrieval. By leveraging recursive hierarchical indexing, dynamic path pruning, and semantic-guided tree navigation, A2X exposes only a small set of highly relevant candidate services at each invocation, effectively decoupling context scarcity from service scale. Experiments demonstrate that A2X achieves a 6.2-point gain in hit rate over full-context baselines while using only one-ninth of the prompt tokens, and outperforms current open-source embedding-based methods by over 20 percentage points.

Abstract

The era of the Internet of Agents (IoA) is taking shape: LLM agents are expected to fulfill user goals by orchestrating fast-growing populations of Model Context Protocol (MCP) servers, Agent-to-Agent (A2A) endpoints, reusable skills, and other LLM-callable services. Yet LLMs face a structural mismatch with this regime: effective context is a scarce resource that does not scale with the number of services. Concatenating thousands of service descriptions into a prompt overflows the context window, and even when the window is large enough, models systematically under-attend to information in the middle of long inputs, the well-documented Lost-in-the-Middle phenomenon. This is fundamentally a question of context management for service discovery. To address this, we propose an LLM-native progressive-disclosure scheme and its concrete instantiation, A2X (Agent-to-Anything service discovery): an LLM-driven pipeline that automatically organizes the registered services into a hierarchical taxonomy and walks it layer by layer at query time, so that every LLM call sees only a small candidate set highly relevant to the user query. This decouples effective-context scarcity from registry size and significantly reduces token consumption while improving retrieval accuracy. Compared to full-context dumping, A2X achieves a 6.2-point Hit Rate gain at one-ninth the prompt-token cost; compared to the state-of-the-art open-source embedding-based baseline, A2X improves Hit Rate by more than 20 points.

Problem

• service discovery
• context management
• LLM agents
• Lost-in-the-Middle
• service taxonomy

Innovation

• LLM-native
• service taxonomy
• progressive disclosure
• context management
• hierarchical retrieval