Test-Time Warmup for Multimodal Large Language Models

2025-09-12

AI Summary

Multimodal large language models (MLLMs) exhibit limited performance on complex cross-modal reasoning tasks, primarily due to insufficient training data scale and lack of fine-grained supervision. To address this, we propose *Test-time Warmup*, a novel test-time adaptation method that dynamically activates and adaptively adjusts model parameters during inference using only weakly supervised auxiliary task data—without requiring additional annotations or parameter-efficient fine-tuning. This is the first work to introduce test-time adaptation into MLLMs, leveraging their standard vision-encoder–connector–LLM architecture to enable instance-level lightweight parameter optimization. Evaluated on Llama-Vision-Instruct, our method achieves relative improvements of 4.03%, 5.28%, and 1.63% on MMMU, VQA-Rad, and GQA benchmarks, respectively. These results demonstrate significantly enhanced adaptability and robustness of MLLMs for intricate multimodal reasoning tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal Large Language Models (MLLMs) hold great promise for advanced reasoning at the intersection of text and images, yet they have not fully realized this potential. MLLMs typically integrate an LLM, a vision encoder, and a connector that maps the vision encoder's embeddings into the LLM's text embedding space. Although each component is pretrained on massive datasets with billions of samples, the entire multimodal model is typically trained on only thousands (or a few million) samples, which can result in weak performance on complex reasoning tasks. To address these shortcomings, instead of relying on extensive labeled datasets for fine-tuning, we propose a Test-Time Warmup method that adapts the MLLM per test instance by leveraging data from weakly supervised auxiliary tasks. With our approach, we observe a relative performance improvement of 4.03% on MMMU, 5.28% on VQA-Rad, and 1.63% on GQA on the Llama-Vision-Instruct model. Our method demonstrates that 'warming up' before inference can enhance MLLMs' robustness across diverse reasoning tasks.

Problem

• Improving multimodal reasoning without extensive fine-tuning
• Addressing weak performance on complex reasoning tasks
• Adapting models per test instance using auxiliary data

Innovation

• Test-Time Warmup adapts per test instance
• Uses weakly supervised auxiliary tasks data
• Enhances robustness without fine-tuning datasets