REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation

2026-06-17

AI Summary

Current static visual question answering (VQA) benchmarks are susceptible to data leakage, leading to inflated model scores that fail to reflect genuine visual understanding capabilities. This work proposes a dynamic evaluation protocol that leverages metadata to regenerate critical visual regions of images on-the-fly during assessment, producing new samples with unambiguous answers, controllable difficulty, and interpretable construction. For the first time, this approach enables online editing of answer-critical regions in VQA evaluation, endowing benchmarks with updatable properties and effectively mitigating training data contamination. Experiments on V*Bench with eight state-of-the-art vision-language models reveal a performance drop of 9.5–18.8 percentage points from original to regenerated samples, substantially exposing models’ reliance on memorization and underscoring the necessity and efficacy of the proposed method.

Abstract

Static visual question answering (VQA) benchmarks age quickly: Once the items leak into training corpora, scores can reflect memorization rather than genuine visual ability, thus obscuring real progress. Rebuilding high-quality benchmarks such as V*Bench requires substantial human annotation, yet each static release can quickly become another leaked artifact. We propose ReKey, a live benchmark protocol that randomly regenerates the answer-bearing local detail, or visual key, in real images at evaluation time. Using human-validated edit slots, ReKey samples fresh instances with new answers, construction-grounded labels, and controlled visual-search difficulty. On V*Bench, the ReKey regenerated benchmark reveals a sharp score jump across eight frontier vision-language models (VLMs): The original items score 9.5--18.8 percentage points higher than the regenerated variants. By making the visual key renewable, ReKey keeps evaluation fresh as models and training data evolve.

Problem

• visual question answering
• benchmark contamination
• evaluation reliability
• static benchmarks
• memorization vs. understanding

Innovation

• visual-key regeneration
• contamination-resilient evaluation
• live benchmark
• vision-language models
• metadata-grounded editing