Efficient Fine-Tuning of Single-Cell Foundation Models Enables Zero-Shot Molecular Perturbation Prediction

2024-12-18

arXiv.org

AI Summary

Predicting cellular transcriptional responses to drug perturbations is a high-dimensional, data-scarce challenge. To address this, we propose Drug-Conditioned Adapter—a lightweight, adapter-based fine-tuning approach built upon a million-scale pre-trained single-cell foundation model. By tuning fewer than 1% of parameters, our method enables zero-shot and few-shot cross-drug and cross-cell-line transcriptional response prediction—the first such framework achieving dual-axis generalization at both molecular and cellular levels on single-cell foundation models. Leveraging multi-level biological representation alignment and few-shot transfer learning, it achieves state-of-the-art performance across diverse generalization settings, significantly improving prediction accuracy for novel drugs and unseen cell lines. Furthermore, we introduce the first standardized evaluation framework for single-cell perturbation prediction, enabling rigorous, reproducible benchmarking.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Multiagent Systems: Other Foundations of Multi Agent Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Predicting transcriptional responses to novel drugs provides a unique opportunity to accelerate biomedical research and advance drug discovery efforts. However, the inherent complexity and high dimensionality of cellular responses, combined with the extremely limited available experimental data, makes the task challenging. In this study, we leverage single-cell foundation models (FMs) pre-trained on tens of millions of single cells, encompassing multiple cell types, states, and disease annotations, to address molecular perturbation prediction. We introduce a drug-conditional adapter that allows efficient fine-tuning by training less than 1% of the original foundation model, thus enabling molecular conditioning while preserving the rich biological representation learned during pre-training. The proposed strategy allows not only the prediction of cellular responses to novel drugs, but also the zero-shot generalization to unseen cell lines. We establish a robust evaluation framework to assess model performance across different generalization tasks, demonstrating state-of-the-art results across all settings, with significant improvements in the few-shot and zero-shot generalization to new cell lines compared to existing baselines.

Problem

• Predicting transcriptional responses to novel drugs efficiently
• Addressing high dimensionality and limited data in cellular responses
• Enabling zero-shot generalization to unseen cell lines

Innovation

• Efficient fine-tuning of single-cell foundation models
• Drug-conditional adapter for molecular perturbation prediction
• Zero-shot generalization to unseen cell lines