Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses

2026-08-18

AI Summary

研究通过对比人类和大型语言模型在评估中的表现，使用探索性因子分析和专家盲评方法，揭示了两者潜在认知结构的差异。

Abstract

The evaluation of large language models (LLMs) relies heavily on human-designed assessments, implicitly assuming that AI and humans employ similar underlying cognitive constructs. Challenging this assumption, we investigate whether the latent factors governing LLM performance carry the same substantive, human-interpretable meaning as the cognitive constructs governing human learners. Using responses from humans and six LLMs across quantitative reasoning and chemistry assessments, we conducted Exploratory Factor Analysis (EFA) separately for both groups. Subject-Matter Experts (SMEs) then blindly evaluated the resulting factor graphs to ascribe pedagogical meaning to the emerged constructs. SMEs successfully interpreted most of the human-derived factors. Conversely, they could not ascribe meaning to any LLM-derived factors in quantitative reasoning and interpreted only half of the LLM factors in chemistry. By combining data-driven EFA with blind expert interpretation, this framework shows that LLMs frequently operate on statistically opaque mechanisms distinct from human reasoning.

Problem

• Large Language Models
• Cognitive Constructs
• Exploratory Factor Analysis
• Human Reasoning

Innovation

• Exploratory Factor Analysis (EFA)
• latent factors
• human-interpretable meaning
• cognitive constructs
• statistically opaque mechanisms