Towards Verifiable Transformers: Solver-Checkable Circuit Explanations

2026-05-21

AI Summary

Existing interpretability methods for Transformers lack formal verification, making it difficult to rigorously establish the functional correctness of identified circuits. This work proposes the first verifiable Transformer framework, which translates local task-specific circuits into bounded propositional logic formulas and employs SMT solvers to formally verify properties such as functional equivalence, edge necessity, and content invariance. For non-encodable operators, the framework introduces a proxy-mediated verification mechanism. It integrates Signed L1 BandNorm, sparsemax attention, and LeakyReLU activation functions, successfully verifying quotation-closing and bracket-type tracking circuits on small symbolic tasks. The approach enables stable training at GPT-2 scale and yields falsifiable symbolic interpretations along with counterexamples through proxy-based verification.

Abstract

Mechanistic interpretability often identifies circuits inside Transformer models, but explanations of those circuits are usually validated through examples, ablations, and manual reasoning. This leaves a gap between finding a plausible circuit and proving what the circuit does. We introduce Verifiable Transformers, a framework for converting task-localized Transformer circuits into bounded, solver-checkable claims. Given a behavior, a finite task domain, and a candidate-token projection, we extract a task circuit and verify properties such as projected functional equivalence, edge necessity, task-relevant invariance, and final-residual robustness. Direct verification encodes the extracted circuit itself into an SMT solver. When a circuit contains operators that are not exactly or tractably encodable, surrogate-mediated verification fits an SMT-encodable surrogate, validates it against the extracted circuit over the bounded domain, and verifies symbolic explanations against the surrogate. We instantiate direct verification with a GPT-style architecture using Signed L1 BandNorm, sparsemax attention, and LeakyReLU. On small symbolic sequence tasks, we train an SMT-representable Transformer, extract sparse circuits for quote closing and bracket type tracking, and exhaustively verify projected functional equivalence, content invariance, edge necessity, and final-residual robustness. At GPT-2 scale, the same operator stack trains stably on OpenWebText, although naive direct SMT verification remains intractable. We also demonstrate surrogate-mediated verification on task-localized circuits with hard-to-encode attention, showing both verified symbolic explanations and solver-generated counterexamples. The goal is not full-model verification, but a concrete path for turning mechanistic circuit explanations into formal propositions that can be proven or refuted.

Problem

• mechanistic interpretability
• circuit verification
• Transformer models
• formal verification
• SMT solvers

Innovation

• Verifiable Transformers
• mechanistic interpretability
• SMT verification
• surrogate-mediated verification
• circuit extraction