How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs

2025-07-18

AI Summary

This study investigates how large language models (LLMs) comprehend temporal semantics in narratives, specifically whether their behavior reflects human-like cognition or sophisticated pattern matching. Method: We introduce “Expert-in-the-Loop Probing”—a novel probing paradigm integrating standardized narrative stimuli, cognitive experimental design, and human baseline controls—to systematically evaluate LLMs across genre identification, temporal sequence representation, and aspectual causal inference. Contribution/Results: We find that LLMs heavily rely on prototypical cues, exhibit low consistency in tense-aspect judgment, and significantly underperform humans in aspect-based causal reasoning. Their temporal semantic representations are unstable and lack cognitive robustness. This work provides the first empirical demonstration of fundamental limitations in LLMs’ narrative temporal understanding and reveals a critical divergence from human cognition. It establishes a reproducible methodological framework and empirically grounded benchmark for evaluating and improving temporal reasoning in language models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) exhibit increasingly sophisticated linguistic capabilities, yet the extent to which these behaviors reflect human-like cognition versus advanced pattern recognition remains an open question. In this study, we investigate how LLMs process the temporal meaning of linguistic aspect in narratives that were previously used in human studies. Using an Expert-in-the-Loop probing pipeline, we conduct a series of targeted experiments to assess whether LLMs construct semantic representations and pragmatic inferences in a human-like manner. Our findings show that LLMs over-rely on prototypicality, produce inconsistent aspectual judgments, and struggle with causal reasoning derived from aspect, raising concerns about their ability to fully comprehend narratives. These results suggest that LLMs process aspect fundamentally differently from humans and lack robust narrative understanding. Beyond these empirical findings, we develop a standardized experimental framework for the reliable assessment of LLMs' cognitive and linguistic capabilities.

Problem

• Assess if LLMs process temporal meaning like humans
• Evaluate LLMs' consistency in aspectual judgments
• Test LLMs' causal reasoning from linguistic aspect

Innovation

• Expert-in-the-Loop probing pipeline
• Targeted experiments on aspectual judgments
• Standardized cognitive assessment framework