Teaching LLMs Program Semantics via Symbolic Execution Traces

2026-05-07

AI Summary

This work addresses the challenge that large language models struggle to effectively detect property violations in program verification, particularly exhibiting significant performance degradation on long programs. The authors propose a novel approach that leverages error traces generated by the symbolic execution engine Soteria as continued pretraining data for the Qwen3-8B model, combined with chain-of-thought reasoning at inference time to enhance semantic understanding of programs. Using only approximately 3,000 such traces, this method improves violation detection accuracy by over 17 percentage points, enabling the 8B-parameter model to surpass a 32B-parameter counterpart without chain-of-thought reasoning. The approach demonstrates balanced performance across five SV-COMP property categories and generalizes to unseen property types, validating the superadditive effect arising from the synergy among error trace semantics, formatting, and chain-of-thought reasoning.

Abstract

We introduce an evaluation framework of 500 C verification tasks across five property types (memory safety, overflow, termination, reachability, data races) built on SV-COMP 2025, and evaluate 14 models across six families. We find that high overall accuracy masks a critical weakness: while most models reliably confirm properties hold, violation detection varies widely and degrades sharply with program length. To close this gap, we train on formal verification artifacts: running the Soteria symbolic execution engine on generic open-source C code and using the resulting traces for continued pretraining of Qwen3-8B. Just ${\sim}$3,000 bug traces combined with chain-of-thought reasoning at inference time improve violation detection by over 17 percentage points, producing one of the most balanced accuracy profiles among evaluated models. On violation detection, the trained 8B model outperforms the 4$\times$ larger Qwen3-32B without thinking and approaches it in overall accuracy. The interaction between trace training and chain-of-thought is superadditive: neither alone provides meaningful gains, but their combination does. Improvements transfer across all five property types, including ones the training traces do not target. Our 28 configurations confirm the gains stem from trace semantics, not code volume, and that trace curation and format matter.

Problem

• program verification
• violation detection
• symbolic execution
• large language models
• property checking

Innovation

• symbolic execution traces
• program semantics
• formal verification
• chain-of-thought reasoning
• LLM fine-tuning