Protein Structure Tokenization via Geometric Byte Pair Encoding

2025-11-13

AI Summary

Protein structure tokenization (PST) lacks interpretable, multi-scale, and architecture-agnostic discretization methods. This paper introduces GeoBPE—the first geometry-aware protein structure tokenization framework—that converts continuous backbone conformations into hierarchical, discrete geometric primitives semantically aligned with CATH functional annotations. Its core innovations include: (i) SE(3)-equivariant differentiable inverse kinematics optimization for conformational fidelity; (ii) geometric k-medoids clustering for structural abstraction; and (iii) a byte-pair encoding–inspired iterative merging strategy to suppress conformational drift. GeoBPE enables joint sequence–structure–function modeling while ensuring full interpretability, multi-resolution control, and cross-architecture transferability. Experiments demonstrate >10× compression, substantially improved data efficiency, and consistent superiority over state-of-the-art methods across 12 downstream tasks. Reconstruction distortion remains tightly bounded at 1.0–1.1, enabling, for the first time, high-fidelity unconditional backbone generation and precise functional family alignment.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Structured Learning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

Protein structure is central to biological function, and enabling multimodal protein models requires joint reasoning over sequence, structure, and function. A key barrier is the lack of principled protein structure tokenizers (PSTs): existing approaches fix token size or rely on continuous vector codebooks, limiting interpretability, multi-scale control, and transfer across architectures. We introduce GeoBPE, a geometry-grounded PST that transforms continuous, noisy, multi-scale backbone conformations into discrete ``sentences'' of geometry while enforcing global constraints. Analogous to byte-pair encoding, GeoBPE generates a hierarchical vocabulary of geometric primitives by iteratively (i) clustering Geo-Pair occurrences with k-medoids to yield a resolution-controllable vocabulary; (ii) quantizing each Geo-Pair to its closest medoid prototype; and (iii) reducing drift through differentiable inverse kinematics that optimizes boundary glue angles under an $mathrm{SE}(3)$ end-frame loss. GeoBPE offers compression ($>$10x reduction in bits-per-residue at similar distortion rate), data efficiency ($>$10x less training data), and generalization (maintains test/train distortion ratio of $1.0-1.1$). It is architecture-agnostic: (a) its hierarchical vocabulary provides a strong inductive bias for coarsening residue-level embeddings from large PLMs into motif- and protein-level representations, consistently outperforming leading PSTs across $12$ tasks and $24$ test splits; (b) paired with a transformer, GeoBPE supports unconditional backbone generation via language modeling; and (c) tokens align with CATH functional families and support expert-interpretable case studies, offering functional meaning absent in prior PSTs. Code is available at https://github.com/shiningsunnyday/PT-BPE/.

Problem

• Developing principled protein structure tokenization for multimodal protein modeling
• Overcoming limitations of fixed token size and continuous vector codebooks
• Enabling interpretable multi-scale protein structure representation and transfer

Innovation

• GeoBPE tokenizes protein structures using geometric byte-pair encoding
• It clusters geometric pairs iteratively with k-medoids for vocabulary
• Differentiable inverse kinematics reduces drift under SE(3) constraints