Engineering Pitfalls in AI Coding Tools: An Empirical Study of Bugs in Claude Code, Codex, and Gemini CLI

2026-03-21

AI Summary

This study addresses the underexplored reliability issues of AI-powered coding tools in real-world engineering practice. Through a large-scale empirical analysis of over 3,800 public bug reports on GitHub concerning Claude-Code, Codex, and Gemini CLI, the authors employ open coding to synthesize issue descriptions, user discussions, and developer responses into a multidimensional defect taxonomy. The findings reveal that more than 67% of reported defects are functional in nature, with 36.9% stemming from API misuse, integration flaws, or configuration errors. The most prevalent symptoms include API errors (18.3%), terminal anomalies (14%), and command failures (12.7%), highlighting critical fragility points during tool invocation and execution. These insights provide empirical grounding and actionable design guidance for developing more robust AI coding assistants.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Humans and AI: Human-AI Collaboration / Human-AI Teaming

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The rapid integration of Large Language Models (LLMs) into software development workflows has given rise to a new class of AI-assisted coding tools, such as Claude-Code, Codex, and Gemini CLIs. While promising significant productivity gains, the engineering process of building these tools, which sit at the complex intersection of traditional software engineering, AI system design, and human-computer interaction, is fraught with unique and poorly understood challenges. This paper presents the first empirical study of engineering pitfalls in building such tools, on a systematic, manual analysis of over 3.8K publicly reported bugs in the open-source repositories of three AI-assisted coding tools (i.e., Claude-Code, Codex, and Gemini CLIs) on GitHub. Specifically, we employ an open-coding methodology to manually examine the issue description, associated user discussions, and developer responses. Through this process, we categorize each bug along multiple dimensions, including bug type, bug location, root cause, and observed symptoms. This fine-grained annotation enables us to characterize common failure patterns and identify recurring engineering challenges. Our results show that more than 67% of the bugs in these tools are related to functionality. In terms of root causes, 36.9% of the bugs stem from API, integration, or configuration errors. Consequently, the most commonly observed symptoms reported by users are API errors (18.3%), terminal problems (14%), and command failures (12.7%). These bugs predominantly affect the tool invocation (37.2%) and command execution (24.7%) stages of the system workflow. Collectively, our findings provide a critical roadmap for developers seeking to design the next generation of reliable and robust AI coding assistants.

Problem

• AI coding tools
• engineering pitfalls
• bugs
• LLM integration
• software reliability

Innovation

• AI coding tools
• empirical study
• engineering pitfalls
• LLM integration
• bug analysis