MolCLIP: A Molecular-Auxiliary CLIP Framework for Identifying Drug Mechanism of Action Based on Time-Lapsed Mitochondrial Images

2025-07-10

AI Summary

Existing MoA identification methods predominantly rely on static cellular images, neglecting dynamic temporal responses of live cells and failing to incorporate drug molecular structural information. To address these limitations, we propose a molecule-augmented CLIP framework that, for the first time, integrates drug molecular representations into a vision-language model to guide spatiotemporal representation learning on mitochondrial time-lapse videos. By leveraging cross-modal alignment and metric learning, our method optimally fuses molecular and video features, jointly modeling drug chemical structure and cellular dynamic phenotypes. Evaluated on MitoDataset, our approach achieves +20.5% mAP for MoA classification and +51.2% mAP for drug identification over state-of-the-art methods. Key contributions include: (1) establishing the first molecule–cell-video bimodal temporal learning paradigm tailored for MoA identification; and (2) designing a molecule-guided CLIP architecture with a dynamic feature aggregation strategy.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Drug Mechanism of Action (MoA) mainly investigates how drug molecules interact with cells, which is crucial for drug discovery and clinical application. Recently, deep learning models have been used to recognize MoA by relying on high-content and fluorescence images of cells exposed to various drugs. However, these methods focus on spatial characteristics while overlooking the temporal dynamics of live cells. Time-lapse imaging is more suitable for observing the cell response to drugs. Additionally, drug molecules can trigger cellular dynamic variations related to specific MoA. This indicates that the drug molecule modality may complement the image counterpart. This paper proposes MolCLIP, the first visual language model to combine microscopic cell video- and molecule-modalities. MolCLIP designs a molecule-auxiliary CLIP framework to guide video features in learning the distribution of the molecular latent space. Furthermore, we integrate a metric learning strategy with MolCLIP to optimize the aggregation of video features. Experimental results on the MitoDataset demonstrate that MolCLIP achieves improvements of 51.2% and 20.5% in mAP for drug identification and MoA recognition, respectively.

Problem

• Identifying drug mechanism via temporal cell imaging dynamics
• Integrating molecular data with cell video for MoA analysis
• Improving drug and MoA recognition accuracy with multimodal learning

Innovation

• Combines microscopic cell video and molecule modalities
• Uses molecule-auxiliary CLIP framework
• Integrates metric learning for feature optimization