Boosting Skeleton-Driven SMT Solver Fuzzing by Leveraging LLM to Produce Formula Generators

2025-08-27

AI Summary

Existing LLM-driven fuzzing approaches for SMT solvers suffer from two critical bottlenecks: nearly half of the generated formulas are syntactically invalid, and iterative LLM invocations incur prohibitive computational overhead. This paper proposes Chimera, the first framework to leverage LLMs for automatically inferring context-free grammars of SMT theories, enabling synthesis of reusable, composable Boolean-term constructors. By integrating grammar-guided skeleton filling, Chimera generates syntactically valid and semantically diverse SMT formulas with only a single LLM query per iteration. The approach unifies grammar extraction, program synthesis, and skeleton-driven fuzz testing. Evaluated on Z3 and cvc5, Chimera discovers 43 vulnerabilities—40 of which have been confirmed and patched—demonstrating substantial improvements in both detection efficiency and practical applicability for industrial-strength SMT solvers.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Satisfiability Modulo Theory (SMT) solvers are foundational to modern systems and programming languages research, providing the foundation for tasks like symbolic execution and automated verification. Because these solvers sit on the critical path, their correctness is essential, and high-quality test formulas are key to uncovering bugs. However, while prior testing techniques performed well on earlier solver versions, they struggle to keep pace with rapidly evolving features. Recent approaches based on Large Language Models (LLMs) show promise in exploring advanced solver capabilities, but two obstacles remain: nearly half of the generated formulas are syntactically invalid, and iterative interactions with the LLMs introduce substantial computational overhead. In this study, we present Chimera, a novel LLM-assisted fuzzing framework that addresses both issues by shifting from direct formula generation to the synthesis of reusable term (i.e., logical expression) generators. Particularly, Chimera uses LLMs to (1) automatically extract context-free grammars (CFGs) for SMT theories, including solver-specific extensions, from documentation, and (2) synthesize composable Boolean term generators that adhere to these grammars. During fuzzing, Chimera populates structural skeletons derived from existing formulas with the terms iteratively produced by the LLM-synthesized generators. This design ensures syntactic validity while promoting semantic diversity. Notably, Chimera requires only one-time LLM interaction investment, dramatically reducing runtime cost. We evaluated Chimera on two leading SMT solvers: Z3 and cvc5. Our experiments show that Chimera has identified 43 confirmed bugs, 40 of which have already been fixed by developers.

Problem

• Generating valid SMT formulas to test solver correctness
• Reducing computational overhead in LLM-assisted fuzzing approaches
• Synthesizing reusable generators instead of direct formula generation

Innovation

• Uses LLM to synthesize reusable term generators
• Extracts context-free grammars from SMT documentation
• Populates structural skeletons with generated terms