Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge

2026-02-11

AI Summary

This work addresses the challenge that multimodal large language models (MLLMs) acting as automatic evaluators struggle to effectively leverage visual examples for prompt optimization due to limited context windows, resulting in insufficient alignment with human judgments. To overcome this, the authors propose BLPO, a bilevel prompt optimization framework that introduces automatic prompt tuning into the MLLM-as-a-Judge paradigm for the first time. The inner loop optimizes image-to-text conversion prompts to preserve evaluation-relevant visual information, while the outer loop jointly refines the judging prompts, enabling efficient cross-modal alignment under strict context-length constraints. Extensive experiments across four datasets and three large language models demonstrate that BLPO significantly improves the consistency between automatic evaluations and human judgments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have become widely adopted as automated judges for evaluating AI-generated content. Despite their success, aligning LLM-based evaluations with human judgments remains challenging. While supervised fine-tuning on human-labeled data can improve alignment, it is costly and inflexible, requiring new training for each task or dataset. Recent progress in auto prompt optimization (APO) offers a more efficient alternative by automatically improving the instructions that guide LLM judges. However, existing APO methods primarily target text-only evaluations and remain underexplored in multimodal settings. In this work, we study auto prompt optimization for multimodal LLM-as-a-judge, particularly for evaluating AI-generated images. We identify a key bottleneck: multimodal models can only process a limited number of visual examples due to context window constraints, which hinders effective trial-and-error prompt refinement. To overcome this, we propose BLPO, a bi-level prompt optimization framework that converts images into textual representations while preserving evaluation-relevant visual cues. Our bi-level optimization approach jointly refines the judge prompt and the I2T prompt to maintain fidelity under limited context budgets. Experiments on four datasets and three LLM judges demonstrate the effectiveness of our method.

Problem

• multimodal LLM-as-a-judge
• auto prompt optimization
• AI-generated image evaluation
• context window constraints
• prompt alignment

Innovation

• Bi-Level Prompt Optimization
• Multimodal LLM-as-a-Judge
• Auto Prompt Optimization
• Image-to-Text Representation
• Context Window Constraints