SWE-smith: Scaling Data for Software Engineering Agents

2025-04-30

AI Summary

In software engineering, training language models faces three key bottlenecks: data scarcity, high construction costs (hundreds of human hours plus TB-scale infrastructure), and limited scale (≤11 repositories, thousands of samples). This paper introduces the first end-to-end automated framework for repository-level training data generation targeting arbitrary Python codebases. It integrates static program analysis, test-driven task synthesis, and sandboxed execution environment construction to fully automate the pipeline from source code to annotated repair tasks. Applied to 128 GitHub repositories, it generates 50K high-quality instances—over an order of magnitude larger than prior work. Fine-tuning SWE-agent-LM-32B on this dataset achieves 40.2% Pass@1 on SWE-bench Verified, setting a new state-of-the-art for open-source models. The complete toolchain and dataset are publicly released, substantially lowering the barrier to AI-for-Software-Engineering research.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite recent progress in Language Models (LMs) for software engineering, collecting training data remains a significant pain point. Existing datasets are small, with at most 1,000s of training instances from 11 or fewer GitHub repositories. The procedures to curate such datasets are often complex, necessitating hundreds of hours of human labor; companion execution environments also take up several terabytes of storage, severely limiting their scalability and usability. To address this pain point, we introduce SWE-smith, a novel pipeline for generating software engineering training data at scale. Given any Python codebase, SWE-smith constructs a corresponding execution environment, then automatically synthesizes 100s to 1,000s of task instances that break existing test(s) in the codebase. Using SWE-smith, we create a dataset of 50k instances sourced from 128 GitHub repositories, an order of magnitude larger than all previous works. We train SWE-agent-LM-32B, achieving 40.2% Pass@1 resolve rate on the SWE-bench Verified benchmark, state of the art among open source models. We open source SWE-smith (collection procedure, task instances, trajectories, models) to lower the barrier of entry for research in LM systems for automated software engineering. All assets available at https://swesmith.com.

Problem

• Limited training data for software engineering LMs
• Complex curation process with high human labor
• Small existing datasets with scalability issues

Innovation

• Automatically synthesizes task instances from codebases
• Constructs execution environments for Python codebases
• Generates large-scale software engineering training data