🤖 AI Summary
Existing repository-level software engineering benchmarks (e.g., SWE-bench) rely on manual annotation, static datasets, and are limited to Python defect repair, lacking multilingual support and scalability. This work introduces the first scalable, multilingual (11 languages) repository-level evaluation benchmark, automatically constructed from real GitHub Pull Requests to yield executable defect repair and feature development tasks. We propose a novel PR-driven, four-stage automation pipeline: programmatic collection, containerized environment synthesis, test oracle extraction, and quality assurance—augmented by prompt-guided failure trajectory synthesis for model training. The benchmark comprises 11,133 instances across 3,971 repositories. On a 1,782-sample subset, Claude-3.5-Sonnet achieves 36.20% pass@10. Fine-tuning on this benchmark significantly improves performance on SWE-bench Multilingual, demonstrating its efficacy for training and evaluation of multilingual code intelligence models.
📝 Abstract
Benchmarks like SWE-bench have standardized the evaluation of Large Language Models (LLMs) on repository-level software engineering tasks. However, these efforts remain limited by manual curation, static datasets, and a focus on Python-based bug fixes. We introduce SWE-Bench++, an automated framework that generates repository-level coding tasks from open-source GitHub projects. Unlike synthetic approaches, our pipeline harvests live pull requests to cover both bug fixes and feature requests across 11 languages. SWE-Bench++ turns GitHub pull requests (PRs) into reproducible, execution-based tasks via four stages: programmatic sourcing, environment synthesis, test oracle extraction, and quality assurance. A final hint-guided trajectory synthesis step converts instances that strong models fail on into training trajectories. Our initial benchmark consists of 11,133 instances from 3,971 repositories across 11 languages. On a subset of 1,782 instances of this benchmark, today's strongest models perform as follows: claude-sonnet-4.5 achieves 36.20% pass@10, gpt-5-2025-08-07 34.57%, gemini/gemini-2.5-pro 24.92%, and gpt-4o 16.89%. We further demonstrate the utility of our dataset by showing that fine-tuning on SWE-Bench++ instances yields measurable improvements on the SWE-bench Multilingual benchmark. SWE-Bench++ provides a scalable, multilingual benchmark for evaluating and improving repository-level code generation.