Synthesis and Verification of Transformer Programs

2026-02-18

AI Summary

This work addresses the problem of automatically verifying and synthesizing C-RASP programs capable of expressing the computational power of Transformers from input-output examples. It presents the first reduction of C-RASP program verification to the verification of Lustre synchronous dataflow programs, enabling efficient verification through modern model checkers and SMT solvers. Additionally, the paper introduces a local search–based synthesis algorithm for C-RASP programs that supports constrained program learning. The approach is evaluated on standard C-RASP benchmarks, demonstrating its effectiveness and successful application to Transformer program optimization and constraint learning from partial specifications.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

C-RASP is a simple programming language that was recently shown to capture concepts expressible by transformers. In this paper, we develop new algorithmic techniques for automatically verifying C-RASPs. To this end, we establish a connection to the verification of synchronous dataflow programs in Lustre, which enables us to exploit state-of-the-art model checkers utilizing highly optimized SMT-solvers. Our second contribution addresses learning a C-RASP program in the first place. To this end, we provide a new algorithm for learning a C-RASP from examples using local search. We demonstrate efficacy of our implementation for benchmarks of C-RASPs in the literature, in particular in connection to the following applications: (1) transformer program optimization, and (2) constrained learning of transformer programs (based on a partial specification).

Problem

• C-RASP
• transformer programs
• program verification
• program synthesis
• constrained learning

Innovation

• C-RASP
• program verification
• transformer synthesis
• local search
• Lustre