Refining LLM-based Directed Test Input Generation via Runtime Value Feedback

2026-08-08

AI Summary

This work addresses the challenge that test inputs generated by large language models (LLMs) often fail to reliably cover target code paths due to limited contextual awareness and uncontrolled reasoning. To overcome this limitation, the authors propose ReDig, a novel framework that introduces runtime value feedback into LLM-driven test generation for the first time, establishing a closed-loop optimization architecture. By monitoring execution traces of previously failed tests, ReDig diagnoses the causes of path non-reachability and iteratively refines subsequent test scripts. Experimental results on Poppler and Libsndfile demonstrate that ReDig effectively leverages runtime feedback to enhance coverage of targeted code lines, significantly improving the reachability and reliability of LLM-generated tests.

Abstract

LLM-based directed input generation techniques have shown promising effectiveness at producing target-reaching test inputs. However, due to the constraint of available code information and inherent unpredictability of LLM inference, reliable directed input generation requires mechanisms to ground the process in observed runtime behavior. We propose ReDig, a runtime feedback-guided refinement framework which adds a control loop around an LLM-based directed test input generation technique to refine the directed input generation with runtime values observed in prior target-missing test executions. In the case studies with Poppler and Libsndfile, we found that ReDig effectively derive runtime value feedback to diagnose why the previous test script failed to reach the target lines, and also effectively leverage given runtime value feedback to refine the test scripts in subsequent steps.

Problem

• LLM-based test input generation
• directed testing
• runtime feedback
• test refinement
• target-reaching inputs

Innovation

• LLM-based testing
• runtime feedback
• directed test input generation
• test script refinement
• ReDig