Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

2025-10-10

AI Summary

Existing prompt optimization methods are confined to the textual modality and thus fail to fully unlock the potential of multimodal large language models (MLLMs). To address this, we introduce the novel task of *multimodal prompt optimization* (MPO), which extends the optimization space to jointly tune textual prompts alongside non-textual modalities—including images, videos, and molecular structures. We propose a unified framework, MPO, featuring (i) an alignment-preserving joint update mechanism that ensures cross-modal semantic consistency, and (ii) a Bayesian prior-guided candidate prompt selection strategy to enhance search efficiency and generalization. Extensive experiments demonstrate that MPO significantly outperforms state-of-the-art text-only prompt optimization methods across diverse multimodal tasks—including visual reasoning, video question answering, and molecular property prediction—establishing a new paradigm for efficient human-MLLM interaction.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) have shown remarkable success, and their multimodal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of multimodal prompts through alignment-preserving updates but also guides the selection process of candidate prompts by leveraging earlier evaluations as priors in a Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.

Problem

• Optimizing multimodal prompts for MLLMs beyond text-only approaches
• Jointly optimizing text and non-text prompts through alignment-preserving updates
• Enhancing MLLM performance across images, videos, and molecular modalities

Innovation

• Joint optimization of multimodal prompts through alignment-preserving updates
• Bayesian-based selection strategy leveraging earlier evaluations as priors
• Unified framework supporting images, videos, and molecules beyond text