MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

2026-07-02

AI Summary

Existing static multimodal evaluation benchmarks are hindered by temporal degradation, data contamination, and high maintenance costs, limiting their ability to sustainably assess vision-language models. This work proposes a multi-agent-driven, automated dynamic evaluation framework that models benchmark evolution as a task-guided dataset construction process. By integrating structured specifications, feedback-controlled real-time data collection, and verifiable question-answer generation, the framework enables efficient and continuous updates. A novel distribution-consistency update strategy is introduced to preserve cross-version comparability while substantially mitigating data contamination risks. Experiments demonstrate that the approach can generate 5.9K high-quality samples in just 1–2 hours at a cost of approximately \$30, effectively maintaining model ranking stability and semantic consistency while significantly reducing memorization signals.

Abstract

Evaluation benchmarks are essential for assessing vision-language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline. Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation with executable reasoning. To maintain cross-version comparability, we introduce a distribution-consistent update strategy that extracts task-related visual patterns from the original benchmark to guide data collection and filtering. Instantiated from MMBench, MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about USD 30 and takes 1-2 hours. Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution. The project is available at https://github.com/PRIS-CV/MMBench-Live.

Problem

• multimodal benchmark
• temporal staleness
• data contamination
• benchmark maintenance
• vision-language models

Innovation

• continuously evolving benchmark
• multi-agent automated pipeline
• distribution-consistent update
• executable reasoning
• multimodal evaluation