Contextualizing Biological Language Models across Modalities via Logit-Space Contrastive Alignment

2026-06-17

AI Summary

Existing biological language models struggle to naturally integrate their predictive distributions into task-specific biological contexts—such as interaction partners or therapeutic interventions—often compromising their native token-level likelihood interface. To address this, this work proposes LOGICA, a framework that introduces contrastive alignment and gated cross-modal adapters in logit space, enabling cross-modal contrastive learning without requiring shared vocabularies or decoders for the first time. LOGICA effectively handles sparse paired data while preserving model interpretability and generative capabilities. The method significantly outperforms current approaches on protein–ligand binding, TCR–peptide activity, and drug-induced resistance prediction tasks, improving the AUC for leave-one-gene-out single-mutation resistance prediction from approximately 0.55 to 0.65.

Abstract

Pretrained biological language models expose per-token probability distributions through masked-token prediction, providing the likelihood interface central to sequence design, variant scoring, and mechanistic interpretation. Yet these distributions are learned from broad unlabeled corpora and are not naturally conditioned on task-specific biological contexts such as interaction partners, cellular environments, or therapeutic interventions. Existing contextual matching methods often distort this interface through pooled embeddings, contrastive latent spaces, or task-specific prediction heads. We introduce LOGICA (Logit-space Contrastive Alignment), a framework for context-conditioned prediction that performs contrastive learning directly in output-logit space. Using gated cross-modal adapters compatible with each model's native token head, LOGICA preserves the pretrained likelihood interface and converts contextualized token log-likelihoods into matching scores. Alignment is defined through context-sensitive token probabilities rather than proximity in a shared embedding space, enabling learning from sparse paired data across models with distinct vocabularies, without a shared tokenizer or decoder. LOGICA is particularly effective for mutation-local variant ranking, where comparisons reduce to context-conditioned likelihoods of mutant tokens at perturbed sites. Across protein--ligand binding, TCR--peptide activity, and drug-conditioned resistance prediction, LOGICA improves over prior state-of-the-art methods, including matched latent-contrastive and conditional MLM baselines, while retaining a token-level interface for interpretation and generation. On held-out-gene single-mutation drug-resistance prediction, LOGICA improves AUC from near-random latent-space baselines of $\sim$0.55 to $\sim$0.65.

Problem

• biological language models
• contextual alignment
• logit-space
• variant scoring
• cross-modal conditioning

Innovation

• logit-space contrastive learning
• context-conditioned prediction
• biological language models
• cross-modal alignment
• token-level likelihood interface