MoRA: On-the-fly Molecule-aware Low-Rank Adaptation Framework for LLM-based Multi-Modal Molecular Assistant

2025-10-14

AI Summary

To address structural feature loss from parameter sharing and catastrophic forgetting induced by fine-tuning in molecular graph–large language model (LLM) integration, this paper proposes an instance-aware dynamic low-rank adaptation mechanism. The pre-trained LLM is frozen, while a graph neural network encodes molecular structures to generate input-specific low-rank adapter weights in real time, which are dynamically injected into the LLM’s feed-forward layers. This approach circumvents limitations of static fine-tuning and fixed adapters, enabling— for the first time—molecular structure-driven personalized parameter generation. It preserves the LLM’s general reasoning capability while substantially enhancing molecular awareness. Experiments demonstrate significant improvements over state-of-the-art baselines: +14.1% exact match rate in chemical reaction prediction and 22% reduction in prediction error for quantum property estimation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Effectively integrating molecular graph structures with Large Language Models (LLMs) is a key challenge in drug discovery. Most existing multi-modal alignment methods typically process these structures by fine-tuning the LLM or adding a static adapter simultaneously. However, these approaches have two main limitations: (1) it optimizes a shared parameter space across all molecular inputs, limiting the model's ability to capture instance-specific structural features; and (2) fine-tuning the LLM for molecular tasks can lead to catastrophic forgetting, undermining its general reasoning capabilities. In this paper, instead of static task-oriented adaptation, we propose an instance-specific parameter space alignment approach for each molecule on-the-fly. To this end, we introduce Molecule-aware Low-Rank Adaptation (MoRA) that produces a unique set of low-rank adaptation weights for each input molecular graph. These weights are then dynamically injected into a frozen LLM, allowing the model to adapt its reasoning to the structure of each molecular input, while preserving the LLM's core knowledge. Extensive experiments demonstrate that on key molecular tasks, such as chemical reaction prediction and molecular captioning, MoRA's instance-specific dynamic adaptation outperforms statically adapted baselines, including a 14.1% relative improvement in reaction prediction exact match and a 22% reduction in error for quantum property prediction. The code is available at https://github.com/jk-sounds/MoRA.

Problem

• Integrating molecular graph structures with Large Language Models for drug discovery
• Addressing limitations of static adapters in capturing instance-specific molecular features
• Preventing catastrophic forgetting while adapting LLMs to molecular tasks

Innovation

• Generates unique low-rank weights per molecule
• Dynamically injects weights into frozen LLM
• Enables instance-specific adaptation without fine-tuning