ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins

2026-03-06

AI Summary

Current protein language models (pLMs) lack a systematic, large-scale benchmark specifically designed for viral proteins, hindering reliable evaluation of their performance in predicting mutational effects and informing antigen selection. To address this gap, this study introduces ViroGym—the first comprehensive benchmark encompassing diverse viruses and phenotypes, integrating 79 deep mutational scanning experiments and influenza neutralization data, along with a novel task for forecasting SARS-CoV-2 prevalent mutations. By aligning in vitro experimental measurements with pLM predictions, the work demonstrates that experimentally validated models can effectively anticipate real-world dominant viral variants. This framework offers a robust foundation for rational vaccine target selection and proactive pandemic forecasting.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Protein language models (pLMs) have shown strong potential in prediction of the functional effects of missense variants in zero-shot settings. Despite this progress, benchmarking pLMs for viral proteins remains limited and systematic strategies for integrating in silico metrics with in vitro validation to guide antigen and target selection are underdeveloped. Here, we introduce ViroGym, a comprehensive benchmark designed to evaluate variant effect prediction in viral proteins and to facilitate selecting rational antigen candidates. We curated 79 deep mutational scanning (DMS) assays encompassing eukaryotic viruses, collectively comprising 552,937 mutated amino acid sequences across 7 distinct phenotypic readouts, and 21 influenza virus neutralisation tasks and a real-world predictive task for SARS-CoV-2. We benchmark well-established pLMs on fitness landscapes, antigenic diversity, and pandemic forecasting to provide a framework for vaccine selection, and show that pLMs selected using in vitro experimental data excel at predicting dominant circulating mutations in real world.

Problem

• viral proteins
• protein language models
• variant effect prediction
• benchmarking
• antigen selection

Innovation

• protein language models
• deep mutational scanning
• viral protein benchmarking
• antigen selection
• variant effect prediction