Can Large Language Models Solve Path Constraints in Symbolic Execution?

2025-11-22

AI Summary

Traditional symbolic execution faces a path constraint solving bottleneck due to SMT solvers’ inability to effectively handle complex data structures and external API calls. Method: This paper pioneers a systematic investigation into replacing SMT solvers with large language models (LLMs) in symbolic execution. We propose an LLM-based approach for path constraint generation and satisfiability classification, integrated into a symbolic execution framework. We construct a comprehensive evaluation benchmark comprising competitive programming problems and real-world open-source projects (e.g., FFmpeg, SQLite), and design an end-to-end automated assessment pipeline to quantitatively measure LLMs’ constraint-solving capabilities across diverse scenarios. Contribution/Results: Experiments show that test cases generated by the LLM achieve 60% precise path coverage; branch coverage improves significantly on real-world repositories; and 32% of previously intractable paths—unsolvable by conventional tools—are successfully resolved. These results validate both the practical efficacy and generalization potential of LLM-augmented symbolic execution.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Symbolic execution is an important software analysis technique which benefits downstream tasks such as software testing and debugging. However, several limitations hinder symbolic execution from application on real-world software. One of the limitations is the inability to solve diverse execution path constraints: traditional symbolic execution based on SMT solvers is difficult to handle execution paths with complex data structures or external API calls. In this paper, we focus on investigating the possibility of adopting large language models (LLM) for path constraint solving instead of traditional solver-based techniques in symbolic execution. We conduct an empirical study to evaluate the ability of LLMs in two types of path constraint solving: generating test inputs to facilitate an execution path, and determining whether a given execution path can be satisfied without triggering any bugs. We build new evaluation pipelines and benchmarks for two tasks: test case generation and path classification, which include data sources from both competition-level programs and real-world repositories. Our experiment results show that state-of-the-art LLMs are able to solve path constraints in both generation and classification tasks, with 60% of generated test cases that accurately cover the given execution path. Moreover, LLMs are capable of improving test coverage by covering execution paths in real-world repositories where traditional symbolic execution tools cannot be applied. These findings highlight the possibility of extending symbolic execution techniques with LLMs in the future to improve the ability and generalizability of symbolic execution.

Problem

• Investigating LLMs' ability to solve symbolic execution path constraints
• Evaluating LLMs for test input generation and path classification tasks
• Assessing LLMs' potential to improve symbolic execution coverage

Innovation

• Using LLMs to solve path constraints in symbolic execution
• Replacing traditional SMT solvers with large language models
• Improving test coverage through LLM-based constraint solving