Cross-Attention and Encoder-Decoder Transformers: A Logical Characterization

2026-05-08

AI Summary

This work addresses the lack of a formal logical characterization for encoder-decoder Transformers employing floating-point soft attention. The authors propose a novel temporal logic that integrates a counting global modality tailored to encoder inputs and a past modality designed for decoder inputs. Coupled with a distributed automaton model, this framework constitutes the first logical characterization suitable for practical floating-point soft attention settings. It naturally accommodates common architectural variants such as masking, establishes an equivalence between Transformers and distributed automata, and extends seamlessly to autoregressive scenarios. The approach demonstrates both robustness to architectural variations and strong expressive power, thereby providing a foundational formalism for reasoning about realistic Transformer models.

Abstract

We give a novel logical characterization of encoder-decoder transformers, the foundational architecture for LLMs that also sees use in various settings that benefit from cross-attention. We study such transformers over text in the practical setting of floating-point numbers and soft-attention, characterizing them with a new temporal logic. This logic extends propositional logic with a counting global modality over the encoder input and a past modality over the decoder input. We also give an additional characterization of such transformers via a type of distributed automata, and show that our results are not limited to the specific choices in the architecture and can account for changes in, e.g., masking. Finally, we discuss encoder-decoder transformers in the autoregressive setting.

Problem

• encoder-decoder transformers
• cross-attention
• logical characterization
• temporal logic
• soft-attention

Innovation

• encoder-decoder transformers
• cross-attention
• temporal logic
• distributed automata
• soft-attention