PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

2026-06-26

AI Summary

This work addresses the discrepancy between high benchmark scores and fragile real-world perceptual capabilities of multimodal models by proposing a fine-grained evaluation framework that bridges the gap between automated metrics and human judgment. Built upon 1,038 high-information-density images and over 12,000 instance-level scoring rules, the framework introduces dual-stream criteria—Must-Right and Easy-Wrong—employs circular peer review to construct gold-standard annotations, and implements a “fail-as-penalty” gated scoring mechanism. Experiments demonstrate that this approach significantly improves alignment with human assessments, exposes critical reliability gaps in dense visual scenes, reveals an 8% perception performance disparity between open- and closed-source models, and validates that the gated metric better reflects human perception than conventional benchmarks.

Abstract

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.

Problem

• multimodal evaluation
• human perception
• benchmark brittleness
• perceptual fidelity
• visual reasoning

Innovation

• PerceptionRubrics
• Gated Scoring
• Atomic Auditing
• Circular Peer-Review
• Multimodal Evaluation