Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers

2026-05-21

AI Summary

This work addresses the lack of theoretical grounding in graph tokenization for Graph Transformers and the unclear impact of different tokenization strategies on model expressivity. The authors systematically analyze three fundamental tokenization approaches—spectral, random-walk, and adjacency-based—and, for the first time, reveal their distinct induced computational depth mechanisms. They establish theoretical lower bounds on structural information loss and non-invertibility arising from these methods. Integrating tools from graph signal processing, representation theory, and deep learning, the study demonstrates on both synthetic and real-world datasets that different tasks inherently favor distinct structural views, and that fusing complementary tokenization schemes significantly enhances performance. The experimental results align closely with theoretical predictions, offering principled guidance for task-driven selection of optimal tokenization strategies.

Abstract

Transformers have become a central architecture for graph learning, but their application to graphs requires first choosing a tokenization: a graph-to-token map that determines which structural information is exposed at the input. In this work, we show that this choice is a fundamental component of transformer expressivity. We examine three tokenizations that serve as building blocks for many existing graph tokenizations: spectral, random-walk, and adjacency tokenizations. We prove that different tokenizations induce distinct depth regimes: the same graph computation may be realizable by a shallow transformer under one tokenization, while requiring substantially larger depth under another. For example, we prove that random-walk tokenization is lossy for any walk length, making it impossible in general to recover the graph from it, and that while spectral tokenization is lossless, it is ill-conditioned for local tasks. We further show that although both random-walk and spectral tokenizations are derived from adjacency information, it is impossible for a limited-depth transformer to convert between tokenization families in general. In particular, we establish lower bounds and impossibility results showing that unfavorable tokenizations may preclude the efficient recovery of more suitable structural representations. Finally, we complement our theory with controlled experiments on synthetic and real-world tasks, validating the predicted separations and showing that different tasks favor different structural views, and combining complementary tokenizations allows the transformer to leverage distinct signals from each representation.

Problem

• graph tokenization
• transformer expressivity
• structural representation
• information loss
• depth regime

Innovation

• graph tokenization
• transformer expressivity
• spectral representation
• random-walk embedding
• structural inductive bias