How Humans and LLMs Organize Conceptual Knowledge: Exploring Subordinate Categories in Italian

2025-05-27

AI Summary

This study investigates cross-linguistic (Italian) alignment between human and large language model (LLM) conceptual knowledge at the subordinate level (e.g., “grizzly bear”), where such comparisons remain underexplored. Method: We introduce the first Italian psycholinguistic dataset of 187 concrete words with subordinate-level exemplar norms, evaluated via a multi-task cognitive framework—exemplar generation, category induction, and typicality rating—comparing human participants against text-based and multimodal LLMs. Results: Overall human–LLM alignment is low, yet significant domain-specific variation emerges: in certain semantic domains, LLM-generated exemplars approximate human typicality distributions. This work establishes the first subordinate-level cognitive alignment paradigm for human–LLM comparison, providing the inaugural cross-lingual empirical foundation and methodological framework for leveraging AI-generated data in cognitive science research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

People can categorize the same entity at multiple taxonomic levels, such as basic (bear), superordinate (animal), and subordinate (grizzly bear). While prior research has focused on basic-level categories, this study is the first attempt to examine the organization of categories by analyzing exemplars produced at the subordinate level. We present a new Italian psycholinguistic dataset of human-generated exemplars for 187 concrete words. We then use these data to evaluate whether textual and vision LLMs produce meaningful exemplars that align with human category organization across three key tasks: exemplar generation, category induction, and typicality judgment. Our findings show a low alignment between humans and LLMs, consistent with previous studies. However, their performance varies notably across different semantic domains. Ultimately, this study highlights both the promises and the constraints of using AI-generated exemplars to support psychological and linguistic research.

Problem

• Examining subordinate-level category organization in humans and LLMs
• Evaluating LLM-generated exemplars alignment with human category structures
• Assessing AI-generated exemplars for psychological and linguistic research

Innovation

• New Italian dataset for subordinate categories
• Evaluate LLMs on human category alignment
• Analyze AI exemplars for psychological research