Metalic: Meta-Learning In-Context with Protein Language Models

2024-10-10

arXiv.org

AI Summary

To address the poor generalization of protein language models (PLMs) under low-data regimes in protein function and biophysical property prediction, this work proposes a meta-learning-driven multi-task transfer framework. It introduces, for the first time, a variant of Model-Agnostic Meta-Learning (MAML) synergistically integrated with in-context learning into PLMs to explicitly model task distributions and enhance cross-task adaptability. Crucially, on tasks unseen during meta-training, only lightweight fine-tuning is required to significantly boost zero-shot and few-shot performance. Evaluated on the low-data ProteinGym benchmark, our method establishes new state-of-the-art results—achieving high prediction accuracy with minimal labeled data, reducing parameter count by 18×, and enabling efficient inference. The core innovations are: (i) a novel synergy between meta-learning and in-context learning; and (ii) a task-adaptive paradigm that achieves strong generalization without requiring meta-training involvement at inference time.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Multiagent Systems: Multiagent Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Predicting the biophysical and functional properties of proteins is essential for in silico protein design. Machine learning has emerged as a promising technique for such prediction tasks. However, the relative scarcity of in vitro annotations means that these models often have little, or no, specific data on the desired fitness prediction task. As a result of limited data, protein language models (PLMs) are typically trained on general protein sequence modeling tasks, and then fine-tuned, or applied zero-shot, to protein fitness prediction. When no task data is available, the models make strong assumptions about the correlation between the protein sequence likelihood and fitness scores. In contrast, we propose meta-learning over a distribution of standard fitness prediction tasks, and demonstrate positive transfer to unseen fitness prediction tasks. Our method, called Metalic (Meta-Learning In-Context), uses in-context learning and fine-tuning, when data is available, to adapt to new tasks. Crucially, fine-tuning enables considerable generalization, even though it is not accounted for during meta-training. Our fine-tuned models achieve strong results with 18 times fewer parameters than state-of-the-art models. Moreover, our method sets a new state-of-the-art in low-data settings on ProteinGym, an established fitness-prediction benchmark. Due to data scarcity, we believe meta-learning will play a pivotal role in advancing protein engineering.

Problem

• Protein Property Prediction
• Data Scarcity
• Machine Learning

Innovation

• Metalic
• Meta-learning
• Protein Property Prediction