Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity

2026-01-26

AI Summary

This study addresses the lack of systematic empirical evidence regarding the real-world impact of coding agents in software development. Leveraging Mining Software Repositories (MSR) methods, it presents the first large-scale analysis of activity traces from large language model–based coding agents on GitHub, systematically identifying their behavioral patterns, potential risks, and effective usage strategies in authentic development environments. The research yields a set of empirically grounded insights concerning optimal timing for agent adoption, reliability concerns, and practical heuristics for deployment. These findings fill a critical gap in the literature, offering actionable guidance for developers and establishing a foundation for future investigations into AI-assisted programming.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

In 2025, coding agents have seen a very rapid adoption. Coding agents leverage Large Language Models (LLMs) in ways that are markedly different from LLM-based code completion, making their study critical. Moreover, unlike LLM-based completion, coding agents leave visible traces in software repositories, enabling the use of MSR techniques to study their impact on SE practices. This paper documents the promises, perils, and heuristics that we have gathered from studying coding agent activity on GitHub.

Problem

• coding agents
• Large Language Models
• software engineering
• mining software repositories
• GitHub

Innovation

• coding agents
• Large Language Models
• Mining Software Repositories
• software engineering practices
• heuristics