ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing

2026-09-04

AI Summary

ProtLingo通过条件记忆和专家路由方法，提高蛋白质语言模型效率，解决大规模Transformer计算成本高且预测突变敏感性不佳的问题。

Abstract

Proteins perform diverse cellular functions, and even single amino-acid substitutions can alter stability, activity, or molecular interactions. Protein language models (PLMs) provide a scalable approach for modeling such sequence--function relationships from unlabeled sequences, but increasing the size of dense Transformer backbones often brings substantial computational cost without consistently improving mutation-sensitive prediction. We introduce ProtLingo, an efficient PLM framework that augments a pretrained single-sequence backbone with conditional local memory and sparse expert routing. ProtLingo maps contextual residue representations into route-specific discrete codes, composes centered local windows into latent $N$-gram addresses, and retrieves reusable residual signals associated with recurring local sequence contexts. In parallel, selected feed-forward blocks are upcycled into sparse Mixture-of-Experts layers with shared and routed experts, enabling residue-dependent computation while activating only a subset of parameters. Experiments on protein fitness prediction, FLIP benchmarks, and supervised contact prediction show that ProtLingo achieves competitive performance with a 150M-scale backbone, including strong parameter efficiency on mutation-effect prediction and preserved long-range structural representations.

Problem

• Protein language models
• mutation-sensitive prediction
• computational cost

Innovation

• conditional local memory
• sparse expert routing
• discrete codes
• N-gram addresses
• Mixture-of-Experts