Phoenix: Safe GitHub Issue Resolution via Multi-Agent LLMs

2026-06-18

AI Summary

This work proposes a multi-agent large language model system designed to automatically and safely resolve GitHub issues without introducing regression errors. The system orchestrates six specialized agents—planning, reproduction, coding, testing, failure analysis, and pull request generation—within an end-to-end pipeline that spans from issue triage to PR submission. It integrates a label-based GitHub Webhook state machine and a baseline-aware test evaluation strategy to guide agent coordination. Crucially, the approach incorporates a seven-layer safety control mechanism—including WAF bypass protection and permission boundary enforcement—and validates fixes through baseline comparison to ensure correctness in real-world deployment. Evaluated on SWE-bench Lite, the system achieves a 75% Oracle solve rate with zero pass-to-pass regressions and demonstrates 100% correctness across 42 real repository issues, resolving challenging cases in an average of 122 seconds.

Abstract

We present Phoenix, a multi-agent LLM system that resolves GitHub issues from triage through pull-request creation, combining seven layered safety controls with a baseline-aware test evaluation strategy. Phoenix decomposes the work across six specialized agents. Planner, reproducer, coder, tester, failure analyst and Pull Request (PR) agent, all coordinated by a label-based GitHub webhook state machine. Every change is checked against a baseline test run before a pull request is opened. On a 24-instance slice of SWE-bench Lite. run on the production webhook path, Phoenix oracle-resolves 75% of instances with no pass-to-pass regressions on successful runs; this curated slice is not directly comparable to full-split leaderboard results, and we discuss the limits of the comparison. A complementary pilot on 42 real issues across 14 repositories yields 100% correctness preservation (CP; mean 122s on the hard tier). Manual inspection shows that about half of the resulting pull requests are well-targeted fixes. The other half place code at incorrect paths, a planner localization limitation we are addressing with retrieval. We also report the deployment failure modes (WAF filtering, token expiry, permission boundaries, flaky CI) that motivated each safety mechanism.

Problem

• GitHub issue resolution
• safety
• correctness preservation
• regression prevention
• automated pull requests

Innovation

• multi-agent LLMs
• GitHub issue resolution
• safety controls
• baseline-aware testing
• automated pull request generation