Do AI Coding Agents Log Like Humans? An Empirical Study

2026-04-10

AI Summary

This study addresses the unclear behavior of AI coding agents in handling non-functional requirements such as software logging and the lack of empirical evaluation regarding the effectiveness of natural language instructions in guiding their logging practices. By systematically analyzing 4,550 agent-submitted pull requests—integrating large-scale open-source repository data, log density metrics, instruction compliance assessments, and manual repair tracking—the work reveals key characteristics of agent logging behavior. Findings indicate that agents modify logs less frequently than humans in 58.4% of repositories; only 4.7% of requests include explicit logging instructions, of which 67% are disregarded; and 72.5% of logging issues require post-hoc human correction. These results demonstrate a dual failure of natural language instructions in enforcing logging norms, underscoring the urgent need for deterministic constraint mechanisms to enhance agents’ capability in generating non-functional code.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Multiagent Systems: Agent Communication
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Software logging is essential for maintaining and debugging complex systems, yet it remains unclear how AI coding agents handle this non-functional requirement. While prior work characterizes human logging practices, the behaviors of AI coding agents and the efficacy of natural language instructions in governing them are unexplored. To address this gap, we conduct an empirical study of 4,550 agentic pull requests across 81 open-source repositories. We compare agent logging patterns against human baselines and analyze the impact of explicit logging instructions. We find that agents change logging less often than humans in 58.4% of repositories, though they exhibit higher log density when they do. Furthermore, explicit logging instructions are rare (4.7%) and ineffective, as agents fail to comply with constructive requests 67% of the time. Finally, we observe that humans perform 72.5% of post-generation log repairs, acting as"silent janitors"who fix logging and observability issues without explicit review feedback. These findings indicate a dual failure in natural language instruction (i.e., scarcity of logging instructions and low agent compliance), suggesting that deterministic guardrails might be necessary to ensure consistent logging practices.

Problem

• AI coding agents
• software logging
• natural language instructions
• empirical study
• log compliance

Innovation

• AI coding agents
• software logging
• empirical study
• natural language instructions
• log repair