Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning

2026-05-11

AI Summary

This work addresses catastrophic forgetting in multimodal large language models during continual learning, as well as the substantial heterogeneity among samples within the same task in terms of visual context, intent, and reasoning requirements. To this end, the authors propose DRAPE, a framework that dynamically generates instance-level continuous soft prompts via cross-modal attention for fine-grained adaptation. DRAPE integrates null-space gradient projection with a CLIP-based prototype routing mechanism, enabling continual instruction tuning without access to task identifiers and while keeping the backbone language model frozen. Moving beyond conventional task-level prompting paradigms, DRAPE achieves state-of-the-art performance on the MCIT benchmark, significantly outperforming existing prompt-based and LoRA-based approaches.

Abstract

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world deployment often requires continual capability expansion across sequential tasks. In such scenarios, Multimodal Continual Instruction Tuning (MCIT) aims to acquire new capabilities while limiting catastrophic forgetting. Existing methods mainly follow a module-composition paradigm: they maintain task-level prompts or LoRA experts and dynamically route or aggregate a subset of them at inference. However, samples within the same task can still differ substantially in visual scenes, question intents, and reasoning demands. This motivates instance-level adaptation to individual query-image pairs rather than only selecting or combining task-level modules. To this end, we propose DRAPE (Dynamic Cross-Modal Prompt Generation), a prompt-learning framework that synthesizes continuous instance-specific soft prompts for MCIT. Instead of selecting prompts from a fixed pool, DRAPE derives prompt queries from the textual instruction and cross-attends to visual patch features, producing query-image conditioned prompts that are prepended to the frozen LLM. To mitigate forgetting during sequential updates, DRAPE applies null-space gradient projection to the shared projector and uses CLIP-based prototype routing for task-label-free generator selection at inference. Extensive experiments on MCIT benchmarks show that DRAPE achieves state-of-the-art performance among representative prompt-based and LoRA-based continual-learning baselines.

Problem

• Multimodal Continual Learning
• Catastrophic Forgetting
• Instance-Level Adaptation
• Instruction Tuning
• Cross-Modal Prompting

Innovation

• Dynamic Prompt Generation
• Cross-Modal Attention
• Continual Learning
• Multimodal Instruction Tuning
• Catastrophic Forgetting Mitigation