Generating and Understanding Tests via Path-Aware Symbolic Execution with LLMs

2025-06-23

AI Summary

This work addresses the challenge that large language models (LLMs) often overlook complex or edge-case execution paths when generating program tests. To tackle this, we propose PALM—a novel system that synergistically integrates symbolic execution with LLMs. PALM performs lightweight, AST-based symbolic path enumeration to generate executable program variants, thereby avoiding the computational overhead of traditional SMT-based constraint solving. Leveraging these variants, we design a path-aware prompting mechanism to guide the LLM in producing high-coverage test cases, augmented with automatically inserted assertions for enhanced verifiability. Furthermore, PALM features an interactive frontend that visualizes path coverage status and supports explainable analysis. To our knowledge, PALM is the first approach to jointly leverage AST-level path enumeration and LLMs for test generation without relying on SMT solvers, significantly improving understanding and coverage of actual execution paths. A user study confirms that PALM effectively enhances developers’ judgment of test adequacy and verification efficiency.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Symbolic execution is a widely used technique for test generation, offering systematic exploration of program paths through constraint solving. However, it is fundamentally constrained by the capability to model the target code including library functions in terms of symbolic constraint and the capability of underlying constraint solvers. As a result, many paths involving complex features remain unanalyzed or insufficiently modeled. Recent advances in large language models (LLMs) have shown promise in generating diverse and valid test inputs. Yet, LLMs lack mechanisms for systematically enumerating program paths and often fail to cover subtle corner cases. We observe that directly prompting an LLM with the full program leads to missed coverage of interesting paths. In this paper, we present PALM, a test generation system that combines symbolic path enumeration with LLM-assisted test generation. PALM statically enumerates possible paths through AST-level analysis and transforms each into an executable variant with embedded assertions that specify the target path. This avoids the need to translate path constraints into SMT formulae, by instead constructing program variants that LLM can interpret. Importantly, PALM is the first to provide an interactive frontend that visualizes path coverage alongside generated tests, assembling tests based on the specific paths they exercise. A user study with 12 participants demonstrates that PALM's frontend helps users better understand path coverage and identify which paths are actually exercised by PALM-generated tests, through verification and visualization of their path profiles.

Problem

• Overcoming symbolic execution limitations in test generation
• Enhancing LLM-based test input diversity and coverage
• Integrating path visualization for user understanding and verification

Innovation

• Combines symbolic path enumeration with LLM-assisted test generation
• Transforms paths into executable variants with embedded assertions
• Provides interactive frontend for path coverage visualization