Verify Implementation Equivalence of Large Models

2026-03-23

AI Summary

Verifying the equivalence of implementations of the same large model across different frameworks is highly challenging due to significant discrepancies in operator decomposition, tensor layouts, and fusion strategies. This work proposes Emerge, a framework that unifies two implementations into a single e-graph representation, infers candidate equivalences guided by runtime values, and automatically synthesizes rewrite rules on demand without manual intervention. By integrating symbolic SMT-based verification with constraint-aware randomized testing, Emerge supports scenarios involving opaque operators. Experimental results demonstrate that Emerge successfully verifies equivalence for correct implementation pairs, detects 10 out of 13 known bugs, and uncovers 8 previously unknown issues confirmed by developers. The automatically generated block-level rewrite rules achieve effectiveness comparable to handcrafted ones.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Machine Learning: Hardware-aware ML

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Verifying whether two implementations of the same large model are equivalent across frameworks is difficult in practice. Even when they realize the same computation, their graphs may differ substantially in operator decomposition, tensor layout, and the use of fused or opaque kernels, making manual rewrite rules hard to build and maintain. We present Emerge, a framework for checking Implementation Equivalence over computation graphs of large-model implementations. Instead of writing rules manually, Emerge represents the two implementations in an e-graph, infers candidate relations from execution values, and synthesizes rewrite rules on demand when existing rules are insufficient. Each synthesized rule is validated using the strongest applicable method, including SMT- based checking for symbolically tractable cases and constraint-aware randomized testing for opaque kernels, and then propagated through e-graph rebuilding to establish larger equivalences. Our current implementation targets inference computation graphs captured from HuggingFace Transformers and vLLM. Our evaluation shows that Emerge establishes equivalence for correct implementation pairs at practical cost, while also providing useful by-products for debugging: it detects 10 of 13 known implementation bugs and uncovers 8 previously unknown implementation issues that were later confirmed by developers. In addition, Emerge synthesizes block-level rules that compare favorably with manually authored ones.

Problem

• implementation equivalence
• large models
• computation graphs
• framework interoperability
• model verification

Innovation

• implementation equivalence
• e-graph
• rewrite rule synthesis
• SMT-based verification
• opaque kernel testing