Basic Category Usage in Vision Language Models

2025-03-16

AI Summary

This study investigates whether open-source vision-language models (VLMs) implicitly acquire human-like basic-level categorization—specifically, differential hierarchical preferences for animate versus inanimate objects and expert-driven shifts in basic-level granularity. Method: Leveraging psycholinguistic paradigms, we conduct zero-shot prompting and category preference analysis to comparatively evaluate Llama 3.2 Vision Instruct (11B) and Molmo 7B-D. Contribution/Results: We provide the first empirical evidence that both models exhibit statistically significant preference for basic-level labels (p < 0.001), with preference patterns highly correlated with human annotations (r = 0.89). Crucially, they successfully replicate the human-specific animate–inanimate effect and expert-induced basic-level shifts. These findings demonstrate that contemporary VLMs spontaneously internalize core aspects of human conceptual structure without explicit training, offering novel evidence for cognitive alignment in multimodal foundation models.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The field of psychology has long recognized a basic level of categorization that humans use when labeling visual stimuli, a term coined by Rosch in 1976. This level of categorization has been found to be used most frequently, to have higher information density, and to aid in visual language tasks with priming in humans. Here, we investigate basic level categorization in two recently released, open-source vision-language models (VLMs). This paper demonstrates that Llama 3.2 Vision Instruct (11B) and Molmo 7B-D both prefer basic level categorization consistent with human behavior. Moreover, the models' preferences are consistent with nuanced human behaviors like the biological versus non-biological basic level effects and the well established expert basic level shift, further suggesting that VLMs acquire cognitive categorization behaviors from the human data on which they are trained.

Problem

• Investigates basic level categorization in vision-language models.
• Compares model categorization preferences with human behavior.
• Explores cognitive behaviors in models trained on human data.

Innovation

• Investigates basic level categorization in VLMs
• Compares Llama 3.2 and Molmo 7B-D models
• Demonstrates human-like categorization in VLMs