Who Writes the Docs in SE 3.0? Agent vs. Human Documentation Pull Requests

2026-01-28

AI Summary

In the context of software engineering’s evolution toward SE 3.0, this study presents the first large-scale empirical investigation into the real-world impact of AI agents on documentation writing and their collaboration patterns with human developers. Leveraging the AIDev dataset, we analyze 1,997 documentation-related pull requests (PRs) through authorship attribution, PR categorization, and manual review behavior analysis. Our findings reveal that AI-generated documentation PRs significantly outnumber those authored by humans and are frequently merged without substantive modifications. This highlights a notable lack of scrutiny in current documentation review practices toward AI-generated content, exposing emerging challenges in ensuring documentation quality within human-AI collaborative workflows. The results offer critical insights for developing reliable and governable AI-assisted software development practices.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Multiagent Systems: Teamwork
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

As software engineering moves toward SE3.0, AI agents are increasingly used to carry out development tasks and contribute changes to software projects. It is therefore important to understand the extent of these contributions and how human developers review and intervene, since these factors shape the risks of delegating work to AI agents. While recent studies have examined how AI agents support software development tasks (e.g., code generation, issue resolution, and PR automation), their role in documentation tasks remains underexplored-even though documentation is widely consumed and shapes how developers understand and use software. Using the AIDev, we analyze 1,997 documentation-related pull requests (PRs) authored by AI agents and human developers, where documentation PRs are those that create or modify project documentation artifacts. We find that AI agents submit substantially more documentation-related PRs than humans in the studied repositories. We further observe that agent-authored documentation edits are typically integrated with little follow-up modification from humans, raising concerns about review practices and the reliability of agent-generated documentation. Overall, while AI agents already contribute substantially to documentation workflows, our results suggest concerns for emerging challenges for documentation quality assurance and human-AI collaboration in SE3.0.

Problem

• AI agents
• software documentation
• pull requests
• human-AI collaboration
• documentation quality

Innovation

• AI agents
• documentation pull requests
• human-AI collaboration
• SE 3.0
• software documentation