Towards an Agent-First Web: Redesigning the Web for AI Agents

2026-06-17

AI Summary

This work addresses the limitations of traditional human-centric web architectures, which hinder legitimate, efficient, and trustworthy interactions for AI agents. The paper proposes the first holistic Agent-First Web framework that treats AI agents as first-class citizens rather than mere crawlers, grounded in ten core design principles. It introduces HTTP metadata for agent identity verification and permission inheritance, an intent-based layered economic model featuring token subscriptions and delegated content mechanisms, and a novel Agent Text Markup Language (ATML) coupled with cryptographic provenance chains to ensure content authenticity. This framework effectively mitigates agent blocking and curbs recursive knowledge distortion, thereby establishing both technical and economic foundations for a collaborative human-agent information ecosystem.

Abstract

The World Wide Web was built on an assumption held for three decades: the primary consumer of web content is a human being. This permeates every layer; its access model presumes human visitors, its economics rest on human attention, and its content targets human perception. The rapid emergence of AI agents as intermediaries between humans and web content invalidates this assumption. Yet the web resists agents through blanket blocking, CAPTCHA-based exclusion, and economic models that treat agent access as extraction rather than legitimate interaction. This paper proposes a principled redesign across three layers. At the access layer, agents acting for humans should inherit equivalent access rights, governed by rate limiting and agent identification metadata in HTTP requests, analogous to browser headers, alongside a dual-layer architecture serving human-readable and agent-optimized content from the same domain. At the economic layer, we propose an intent-based tier framework grounded in the agent-as-human-proxy principle: an agent's economic obligation mirrors that of the human it represents. A token-based subscription model meters content in tokens rather than pageviews, alongside a commissioned content economy anchoring AI content production in human intentionality. At the content layer, we identify epistemic recursion, the self-referential loop in which AI-generated content is consumed by agents to produce further content, progressively detaching web knowledge from human ground truth. We propose the Agent Text Markup Language (ATML), a four-level human supervision tier model, and a cryptographic provenance chain to counter this threat. Together these constitute ten design principles for an agent-first internet, one in which agents are first-class citizens whose integration requires renegotiating the web's foundational social contract across access, economics, and content.

Problem

• AI agents
• World Wide Web
• access control
• economic models
• content authenticity

Innovation

• AI agents
• Agent-First Web
• ATML
• epistemic recursion
• intent-based economics