Context Engineering for AI Agents in Open-Source Software

2025-10-24

AI Summary

This study addresses context engineering for AI agents in open-source projects: specifically, how to structure project-specific knowledge—such as architecture, interfaces, and coding conventions—to improve code generation quality. We conduct the first systematic empirical analysis of 466 open-source AI configuration files (e.g., AGENTS.md), augmented with version-history tracking to characterize their evolution. Our findings reveal that current configurations lack standardized structure, exhibit high syntactic and semantic heterogeneity, and evolve via incremental expansion and collaborative maintenance. Our contributions are threefold: (1) a characterization of structural diversity and evolutionary patterns in real-world AI configuration files; (2) identification of key design dimensions governing contextual effectiveness; and (3) a novel paradigm that links structural optimization of context to generation quality improvement—providing empirical grounding and actionable design guidelines for standardizing AI-ready open-source infrastructure.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Multiagent Systems: Agent/AI Theories and Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

GenAI-based coding assistants have disrupted software development. Their next generation is agent-based, operating with more autonomy and potentially without human oversight. One challenge is to provide AI agents with sufficient context about the software projects they operate in. Like humans, AI agents require contextual information to develop solutions that are in line with the target architecture, interface specifications, coding guidelines, standard workflows, and other project-specific policies. Popular AI agents for software development (e.g., Claude Code) advocate for maintaining tool-specific version-controlled Markdown files that cover aspects such as the project structure, building and testing, or code style. The content of these files is automatically added to each prompt. AGENTS.md has emerged as a potential standard that consolidates tool-specific formats. However, little is known about whether and how developers adopt this format. Therefore, in this paper, we present the results of a preliminary study investigating the adoption of AI configuration files in 466 open-source software projects, what information developers provide in these files, how they present that information, and how they evolve over time. Our findings indicate that there is no established structure yet, and that there is a lot of variation in terms of how context is provided (descriptive, prescriptive, prohibitive, explanatory, conditional). We see great potential in studying which modifications in structure or presentation can positively affect the quality of the generated content. Finally, our analysis of commits that have modified AGENTS.md files provides first insights into how projects continuously extend and maintain these files. We conclude the paper by outlining how the adoption of AI configuration files in provides a unique opportunity to study real-world prompt and context engineering.

Problem

• Investigating adoption of AI configuration files in open-source software projects
• Analyzing content and structure variations in AGENTS.md files
• Studying evolution and maintenance of context engineering practices

Innovation

• Uses AGENTS.md files for AI agent context
• Studies adoption in 466 open-source software projects
• Analyzes evolution of AI configuration files