🤖 AI Summary
This study addresses the challenge of attribute misbinding in large vision-language models within dense homogeneous scenes, where existing metrics prove inadequate. We formally define the DSCAM task and construct InstaBind-Lite, a controlled benchmark accompanied by a specialized evaluation framework. Through fine-grained instance annotation and multi-level question-answering design, this work enables quantitative assessment of attribute transfer while exposing critical blind spots in traditional evaluations. Experiments reveal misbinding rates of 19.84% for open-source models and 7.55% for API-based models, precisely localizing error sources. Ultimately, this research establishes a novel, traceable evaluation paradigm for assessing fine-grained attribute binding capabilities in large multimodal models.
📝 Abstract
Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance. Generic visual-question-answering accuracy marks the response as wrong, while object-hallucination metrics may regard both the object and attribute as image-supported; neither reveals the transfer. This study formalizes this blind spot as Dense Same-Class Attribute Misbinding (DSCAM) and presents InstaBind-Lite, a controlled benchmark that makes it directly measurable. Its 524 images contain 529 curated groups of 3-6 same-class entities, 1773 boxed instances, ordered neighbors, distinguishable color-like attributes, and four complementary question levels, yielding 9580 deterministically evaluated questions. Unlike existing protocols, source-instance annotations separate unsupported generation and recognition failure from an attribute copied from another visible entity. Binding-specific metrics further quantify transfer frequency, adjacency, ordinal distance, and intervention effects. Across five open-source and two commercial/API models, the open-source systems average 19.84% Misbinding Rate and the API systems 7.55%; these errors are hidden by aggregate accuracy. Among identifiable transfers, 80.70% and 81.51%, respectively, originate from adjacent instances. Localization and instance-first interventions help selected models but are not universal remedies. InstaBind-Lite therefore turns previously undifferentiated wrong answers into source-identifiable failure categories and tests a reliability dimension that conventional benchmarks cannot determine: whether a model knows not only what is visible, but which instance owns each attribute.