Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking

2026-05-13

AI Summary

This work addresses the challenge of checkpoint selection for multimodal large language models in dense OCR scenarios, where subtle performance differences and evaluation noise hinder reliable decisions. Framing the problem as robust decision-making under evaluation uncertainty, the authors propose a multi-stage ranking framework that integrates LLM-based proxy judgments, subsampling-based confidence estimation, and a percentile scoring mechanism. By employing a stability-aware ranking strategy, the method effectively captures the performance distribution and penalizes tail-end failures. The approach significantly enhances the robustness and deployment consistency of checkpoint selection and further uncovers the critical influence of OCR readability on evaluation validity.

Abstract

Checkpoint selection for multimodal large language models (MLLMs) presents significant challenges when performance differentials are marginal and evaluation signals are prone to noise. Existing methodologies rely heavily on static benchmarks or pointwise scoring, which frequently misalign with in-the-wild usage and lack robust uncertainty estimation, particularly in OCR-heavy scenarios. In this work, we formulate checkpoint selection as a robust decision problem under evaluation uncertainty. We propose a multi-stage framework that integrates curated real-world data, structured LLM-based judgment, and multi-stage ranking protocols. The evaluation system orchestrates progressive refinement via pointwise filtering, listwise ranking, and pairwise comparison. To enhance reliability, we introduce subsampling-based confidence estimation and a percentile-based scoring formulation that captures distributional characteristics while penalizing tail failures. Furthermore, we demonstrate that data quality, specifically OCR readability, is a critical determinant of evaluation validity.

Problem

• checkpoint selection
• multimodal LLMs
• evaluation uncertainty
• OCR-heavy scenarios
• robust decision

Innovation

• checkpoint selection
• multimodal LLMs
• agentic evaluation
• stability-aware ranking
• uncertainty estimation