Internal APIs Are All You Need: Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures

2026-04-01

AI Summary

Existing autonomous agents rely on browser automation for web interaction, suffering from inefficiency, poor robustness, and redundant operations. This work proposes Unbrowse, a system that systematically leverages internal website APIs—termed Shadow APIs—as a replacement for conventional browser automation, constructing a shared routing graph to enable efficient and collaborative agent interactions. Key innovations include passively learning routes from real user traffic, a three-tier execution pipeline (local cache, shared graph, browser fallback), a shared discovery mechanism, and a three-layer micropayment model based on the x402 protocol to support voluntary participation and a self-correcting architecture. Experiments across 94 websites demonstrate that, with sufficient caching, Unbrowse achieves a mean response time of 950 ms—3.6× faster than Playwright on average (5.4× at the median)—with some requests completing in under 100 ms.

Technology Category

• Multiagent Systems: Mechanism Design
• Search and Optimization: Distributed Search
• Application Domains: Web

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Sustainability of Web economics
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Autonomous agents increasingly interact with the web, yet most websites remain designed for human browsers -- a fundamental mismatch that the emerging ``Agentic Web'' must resolve. Agents must repeatedly browse pages, inspect DOMs, and reverse-engineer callable routes -- a process that is slow, brittle, and redundantly repeated across agents. We observe that every modern website already exposes internal APIs (sometimes called \emph{shadow APIs}) behind its user interface -- first-party endpoints that power the site's own functionality. We present Unbrowse, a shared route graph that transforms browser-based route discovery into a collectively maintained index of these callable first-party interfaces. The system passively learns routes from real browsing traffic and serves cached routes via direct API calls. In a single-host live-web benchmark of equivalent information-retrieval tasks across 94 domains, fully warmed cached execution averaged 950\,ms versus 3{,}404\,ms for Playwright browser automation (3.6$\times$ mean speedup, 5.4$\times$ median), with well-cached routes completing in under 100\,ms. A three-path execution model -- local cache, shared graph, or browser fallback -- ensures the system is voluntary and self-correcting. A three-tier micropayment model via the x402 protocol charges per-query search fees for graph lookups (Tier~3), a one-time install fee for discovery documentation (Tier~1), and optional per-execution fees for site owners who opt in (Tier~2). All tiers are grounded in a necessary condition for rational adoption: an agent uses the shared graph only when the total fee is lower than the expected cost of browser rediscovery.

Problem

• Autonomous agents
• Shadow APIs
• Browser automation
• API discovery
• Agentic Web

Innovation

• Shadow APIs
• Shared Route Graph
• Agentic Web
• Browser Automation
• x402 Micropayments