Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests

2026-01-08

arXiv.org

AI Summary

This work addresses the critical issue of inconsistency between AI-generated pull request (PR) descriptions and the actual code changes, which undermines developer trust in coding agents. The study introduces the first metric for PR Message–Code Inconsistency (PR-MCI) and constructs a large-scale annotated dataset comprising 23,247 PRs to systematically evaluate outputs from five categories of AI coding agents. Through human annotation, statistical testing, and quantitative alignment analysis, the authors identify eight distinct inconsistency patterns. They find that 1.7% of PRs exhibit high PR-MCI, with 45.4% falsely claiming unimplemented changes. Such inconsistent PRs suffer a 51.7% lower acceptance rate and take 3.5 times longer to merge, significantly impeding collaborative development efficiency.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Multiagent Systems: Agent Communication
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Pull request (PR) descriptions generated by AI coding agents are the primary channel for communicating code changes to human reviewers. However, the alignment between these messages and the actual changes remains unexplored, raising concerns about the trustworthiness of AI agents. To fill this gap, we analyzed 23,247 agentic PRs across five agents using PR message-code inconsistency (PR-MCI). We contributed 974 manually annotated PRs, found 406 PRs (1.7%) exhibited high PR-MCI, and identified eight PR-MCI types, revealing that"descriptions claim unimplemented changes"was the most common issue (45.4%). Statistical tests confirmed that high-MCI PRs had 51.7% lower acceptance rates (28.3% vs. 80.0%) and took 3.5 times longer to merge (55.8 vs. 16.0 hours). Our findings suggest that unreliable PR descriptions undermine trust in AI agents, highlighting the need for PR-MCI verification mechanisms and improved PR generation to enable trustworthy human-AI collaboration.

Problem

• message-code inconsistency
• AI coding agent
• pull request
• trustworthiness
• human-AI collaboration

Innovation

• PR message-code inconsistency
• AI coding agents
• pull request description
• trustworthy AI collaboration
• manual annotation