The Faithfulness Gap: Certifying Semantic Equivalence Between Natural-Language and Formal Mathematical Statements

2026-06-15

AI Summary

This work addresses the “faithfulness gap” in automatic formalization, where natural language mathematical statements and their formal counterparts may exhibit semantic inconsistencies—even when the formal statement is well-typed and provable, it might encode an incorrect theorem. To verify semantic faithfulness, the authors propose the Bidirectional Provable Fingerprinting (BPF) framework, which characterizes candidate formal statements through their forward and backward inferential neighborhoods within the ambient theory and matches these against probes derived from the natural language input. Key innovations include counterfactual probe generation, continuous scoring via equivalence spectra, adaptive probe budget allocation, and faithfulness-guided decoding, accompanied by theoretical guarantees including a drift detection theorem and PAC-faithfulness bounds. Experiments on DriftBench show that BPF+CPG detects 89.6% of drifted formalizations with only a 3.0% false positive rate—substantially outperforming baselines—and that Faithfulness-Guided Decoding reduces drift rates in state-of-the-art formalizers by 47%.

Abstract

Autoformalization, translating natural-language mathematics into formal proof assistants, is bottlenecked not by translation fluency but by \emph{faithfulness}: a formal statement can typecheck and be provable, yet still encode a different theorem than the source intended. We introduce \emph{Bidirectional Provability Fingerprinting} (\bpf{}), a framework that certifies faithfulness by characterizing each candidate through its forward and backward consequence neighborhoods in the ambient theory and matching these against probes derived from the natural-language statement. We further introduce four novel components: (i) \emph{Counterfactual Probe Generation} (\cpg{}), a contrastive procedure that synthesizes probes targeting specific drift directions; (ii) the \emph{Equivalence Spectrum}, a continuous faithfulness score that replaces brittle binary verdicts; (iii) \emph{Adaptive Probe Budget Allocation} (\apba{}), an information-theoretic budget router; and (iv) \emph{Faithfulness-Guided Decoding} (\fgd{}), which uses \bpf{} signals as a reward during autoformalization. We prove a \emph{drift detection theorem} and a \emph{PAC-faithfulness} result establishing that the equivalence class of a natural language statement is learnable from $\mathcal{O}(\log(1/δ)/\varepsilon)$ probes under mild assumptions. We release \driftbench{}, a benchmark of $2{,}183$ NL/Lean~4 pairs with controlled drift labels across six subfields of mathlib4. \bpf{}\,+\,\cpg{} detects $89.6\%$ of drifted formalizations at a $3.0\%$ false-positive rate-against $41.2\%$ for typecheck and $63.3\%$ for LLM-judge baselines, and \fgd{} reduces the rate at which a state-of-the-art autoformalizer emits drifted statements by $47\%$. https://pmlrbd.github.io/BPF/

Problem

• faithfulness
• autoformalization
• semantic equivalence
• formal mathematics
• natural-language mathematics

Innovation

• Bidirectional Provability Fingerprinting
• Faithfulness Certification
• Counterfactual Probe Generation
• Equivalence Spectrum
• Autoformalization