Otter: Generating Tests from Issues to Validate SWE Patches

2025-02-07

AI Summary

In test-first development and SWE-agent verification, a key challenge is generating executable, verifiable test cases from natural-language problem descriptions to validate code patches that do not yet exist. Method: We propose the first method that generates high-fidelity, repair-oriented test cases solely from problem text, without access to source code or candidate patches. Our approach introduces an LLM-driven self-reflective action planning framework that integrates rule-based static analysis with semantic consistency checking to overcome critical limitations of purely generative test generation—namely, poor executability and logical incorrectness. Contribution/Results: Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines in test quality (e.g., pass rate, validity, and relevance) and substantially improves both correctness and robustness of downstream patch-generation systems. This work establishes a new paradigm for test synthesis in pre-implementation validation scenarios.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Multiagent Systems: Multiagent Planning

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch that resolves the issue. In this work, we focus on the scenario where the code patch does not exist yet. This approach supports two major use-cases. First, it supports TDD (test-driven development), the discipline of"test first, write code later"that has well-documented benefits for human software engineers. Second, it also validates SWE (software engineering) agents, which generate code patches for resolving issues. This paper introduces Otter, an LLM-based solution for generating tests from issues. Otter augments LLMs with rule-based analysis to check and repair their outputs, and introduces a novel self-reflective action planning stage. Experiments show Otter outperforming state-of-the-art systems for generating tests from issues, in addition to enhancing systems that generate patches from issues. We hope that Otter helps make developers more productive at resolving issues and leads to more robust, well-tested code.

Problem

• Generates tests from software issues
• Supports test-driven development practices
• Validates software engineering agents' patches

Innovation

• LLM-based test generation
• rule-based output analysis
• self-reflective action planning