Transformer Encoder Satisfiability: Complexity and Impact on Formal Reasoning

2024-05-28

AI Summary

This work investigates the satisfiability problem (trSAT) for Transformer encoders (TEs), a core computational challenge in formal verification and model interpretability. We first prove, for general TEs, that trSAT is undecidable—a foundational result establishing inherent limits on automated reasoning about unbounded-precision TE behavior. Building on this, we systematically construct a complexity hierarchy for TE satisfiability, demonstrating that imposing practical quantization constraints—e.g., fixed-point precision—restores decidability. Under such realistic assumptions, we rigorously establish that trSAT is NEXPTIME-hard and NEXPTIME-complete, thereby determining a tight complexity bound. Methodologically, our approach integrates formal language theory, computability analysis, and computational complexity theory, while introducing novel abstractions: quantized neural network modeling and attention mechanism formalization. This work provides the first deep theoretical foundation characterizing the logical properties of TEs, enabling principled formal verification of transformer-based AI systems.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We analyse the complexity of the satisfiability problem, or similarly feasibility problem, (trSAT) for transformer encoders (TE), which naturally occurs in formal verification or interpretation, collectively referred to as formal reasoning. We find that trSAT is undecidable when considering TE as they are commonly studied in the expressiveness community. Furthermore, we identify practical scenarios where trSAT is decidable and establish corresponding complexity bounds. Beyond trivial cases, we find that quantized TE, those restricted by fixed-width arithmetic, lead to the decidability of trSAT due to their limited attention capabilities. However, the problem remains difficult, as we establish scenarios where trSAT is NEXPTIME-hard and others where it is solvable in NEXPTIME for quantized TE. To complement our complexity results, we place our findings and their implications in the broader context of formal reasoning.

Problem

• Analyze transformer encoder satisfiability complexity.
• Identify decidable scenarios for trSAT.
• Establish NEXPTIME-hardness for quantized TE.

Innovation

• Transformer Encoder complexity analysis
• Quantized TE decidability scenarios
• NEXPTIME-hard problem identification