Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs

2026-06-18

AI Summary

Existing continual learning benchmarks for multimodal large language models (MLLM-CL) suffer from highly separable tasks and fixed sequences, limiting their ability to evaluate forward transfer and robustness in realistic scenarios. This work challenges the prevailing assumption that routing mechanisms must rely on large, trainable models and introduces RePRo, a lightweight prototype-based routing method that requires neither training nor replay. By leveraging frozen pretrained features and task-specific prototypes, RePRo constructs an efficient routing mechanism. Experiments demonstrate that RePRo matches the performance of MR-LoRA while substantially reducing computational costs. Ablation studies further reveal that shared experts provide no meaningful benefit for continual learning and indicate that MLLM-CL primarily rewards isolated task learning rather than genuine continual transfer capabilities.

Abstract

Continual adaptation is essential for multimodal large language models (MLLMs) deployed across evolving domains, but the state-of-the-art MR-LoRA method highly relies on the assumption that a MLLM-based router is necessary to process complex multimodal inputs. This paper revisits this claim on the MLLM-CL benchmark and argues for two claims. \textbf{First}, routing does not require an MLLM: a simple training-free, replay-free ptotypical routing method (\textsc{RePRo}), uses frozen pretrained features and task prototypes to match the MLLM-based router of MR-LoRA at far lower computational cost. \textbf{Second}, shared experts do not improve continual learning for MLLMs, despite their theoretical appeal. We show that these findings arise from two structural limitations of MLLM-CL: (1) its tasks are \textbf{highly separable} in representation space, and (2) its fixed task order makes conclusions \textbf{sensitive to a single curriculum} rather than robust across diverse continual-learning trajectories. As a result, the benchmark primarily rewards learning in isolation rather than genuine continual transfer. This motivates a new design for future benchmarks of continual MLLM learning, with overlapping task manifolds, multiple task orders, fine-grained domain shifts, and evaluation protocols that reward forward transfer as well as retention.

Problem

• continual learning
• multimodal large language models
• benchmark evaluation
• task separability
• curriculum sensitivity

Innovation

• continual learning
• multimodal large language models
• prototype-based routing
• benchmark design
• forward transfer