VeryTrace: Verifying Reasoning Traces through Compilable Formalism and Structured Verification

2026-06-23

AI Summary

This work addresses the challenge of silent error propagation in multi-step reasoning, where early logical mistakes or hallucinations often lead large language models to produce confidently incorrect conclusions. The authors propose a zero-shot verification and repair framework that formalizes natural language reasoning traces into a structured domain-specific language (DSL), explicitly encoding step dependencies, executable quantitative expressions, and deductive structures. By integrating deterministic checks—such as computational correctness and constraint satisfaction—with semantic auditing via large language models, the approach enables training-free, step-level error detection and correction. Evaluated on mathematical reasoning, robotic planning, and kinship inference tasks, the method substantially outperforms zero-shot baselines, significantly enhancing the reasoning accuracy of mainstream large models without requiring domain-specific data or exemplars.

Abstract

Multi-step reasoning with Chain-of-Thought (CoT) prompting remains fragile: logical errors or hallucinations in early steps silently propagate, producing confident but incorrect conclusions. This paper presents VeryTrace, a zero-shot verification-and-repair framework that formalizes natural-language reasoning traces into a structured, compilable representation. VeryTrace introduces a Domain-Specific Language (DSL) that (i) makes step dependencies explicit, (ii) mechanizes quantitative content as executable expressions, and (iii) structures semantic inferences via deduction schemas. Our hybrid verifier combines deterministic checks for computational correctness, dependency resolution, and constraint satisfaction with targeted LLM audits for non-mechanizable semantic judgments, enabling step-level error localization and repair. Across three diverse domains-competition mathematics (AIME 2025), robotics planning (LLM-BabyBench), and kinship reasoning (CLUTRR), VeryTrace improves accuracy over zero-shot baselines on state-of-the-art LLMs without requiring domain-specific training or in-context examples, demonstrating that formalized trace verification achieves both precision and generalization.

Problem

• Chain-of-Thought
• reasoning trace
• logical error
• hallucination
• multi-step reasoning

Innovation

• Chain-of-Thought verification
• Domain-Specific Language (DSL)
• structured reasoning trace
• zero-shot repair
• hybrid verification