CogLM: Tracking Cognitive Development of Large Language Models

2024-08-17

arXiv.org

AI Summary

This study systematically evaluates the developmental trajectory and underlying mechanisms of cognitive capabilities in large language models (LLMs). Grounded in Piaget’s theory of cognitive development, we introduce CogLM—the first dedicated benchmark for assessing LLM cognition—comprising 10 cognitive dimensions and 1,220 expert-crafted items. Methodologically, we pioneer the integration of developmental psychology into LLM evaluation, combining multi-dimensional human annotation, cross-model large-scale assessment, and causal attribution analysis. Key contributions include: (1) empirical evidence that GPT-4’s cognitive proficiency approximates that of a 20-year-old human; (2) validation that parameter scale and optimization objectives are primary drivers of cognitive evolution; and (3) discovery of stepwise cognitive growth in LLMs, strongly correlated with downstream task performance. This work establishes a novel theoretical framework and empirical foundation for interpretable and controllable LLM development.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Piaget's Theory of Cognitive Development (PTC) posits that the development of cognitive levels forms the foundation for human learning across various abilities. As Large Language Models (LLMs) have recently shown remarkable abilities across a wide variety of tasks, we are curious about the cognitive levels of current LLMs: to what extent they have developed and how this development has been achieved. To this end, we construct a benchmark CogLM (Cognitive Ability Evaluation for Language Model) based on PTC to assess the cognitive levels of LLMs. CogLM comprises 1,220 questions spanning 10 cognitive abilities crafted by more than 20 human experts, providing a comprehensive testbed for the cognitive levels of LLMs. Through extensive experiments across multiple mainstream LLMs with CogLM, we find that: (1) Human-like cognitive abilities have emerged in advanced LLMs (GPT-4), comparable to those of a 20-year-old human. (2) The parameter size and optimization objective are two key factors affecting the cognitive levels of LLMs. (3) The performance on downstream tasks is positively correlated with the level of cognitive abilities. These findings fill the gap in research on the cognitive abilities of LLMs, tracing the development of LLMs from a cognitive perspective and guiding the future direction of their evolution.

Problem

• Large Language Models
• Cognitive Capabilities
• Model Scalability

Innovation

• CogLM
• Cognitive Abilities
• Large Language Models