Cognitive networks highlight differences and similarities in the STEM mindsets of human and LLM-simulated trainees, experts and academics

2025-02-26

AI Summary

This study investigates similarities and differences in STEM-domain cognitive structures between humans and large language models (LLMs), specifically examining how domain expertise shapes conceptual organization. Method: We construct behavioral-forma mentis networks (BFMNs) from semantic association data, comparing 177 human participants (novices, experts, scholars) with 177 GPT-3.5-generated “artificial humans” simulating equivalent response patterns. Contribution/Results: Human BFMNs exhibit significantly higher clustering coefficients—indicating stronger triadic closure and conceptual integration—particularly among domain experts. In contrast, GPT-3.5 BFMNs are sparser, revealing structural limitations: deficient conceptual closure and absence of experiential grounding. Notably, we identify mathematics as consistently neutral-to-positive in valence across both human and LLM networks—a first empirical demonstration of shared affective semantics in STEM. These findings establish a novel paradigm for cross-system cognitive comparison and provide an empirically grounded benchmark for assessing AI-human心智 comparability in scientific reasoning.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Application Domains: Humanities & Computational Social Science
• Humans and AI: Learning Human Values and Preferences

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Understanding attitudes towards STEM means quantifying the cognitive and emotional ways in which individuals, and potentially large language models too, conceptualise such subjects. This study uses behavioural forma mentis networks (BFMNs) to investigate the STEM-focused mindset, i.e. ways of associating and perceiving ideas, of 177 human participants and 177 artificial humans simulated by GPT-3.5. Participants were split in 3 groups - trainees, experts and academics - to compare the influence of expertise level on their mindset. The results revealed that human forma mentis networks exhibited significantly higher clustering coefficients compared to GPT-3.5, indicating that human mindsets displayed a tendency to form and close triads of conceptual associations while recollecting STEM ideas. Human experts, in particular, demonstrated robust clustering coefficients, reflecting better integration of STEM concepts into their cognitive networks. In contrast, GPT-3.5 produced sparser mindsets. Furthermore, both human and GPT mindsets framed mathematics in neutral or positive terms, differently from STEM high schoolers, researchers and other large language models sampled in other works. This research contributes to understanding how mindset structure can provide cognitive insights about memory structure and machine limitations.

Problem

• Compare STEM mindsets of humans and GPT-3.5
• Analyze cognitive network clustering coefficients
• Evaluate expertise influence on STEM conceptualization

Innovation

• Behavioral Forma Mentis Networks
• GPT-3.5 Simulated Humans
• Clustering Coefficients Analysis