A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models

2026-04-29

AI Summary

This study investigates whether large language models adopt human-like sentence comprehension strategies—particularly the trade-off between memory and processing—under cognitive resource constraints. To this end, we introduce a dual-task paradigm that integrates arithmetic computation with sentence acceptability judgment, enabling direct manipulation of the model’s allocated cognitive resources for the first time. Experiments on GPT-4o, o3-mini, and o4-mini reveal that under dual-task conditions, the accuracy gap between judging plausible versus implausible sentences significantly widens, indicating a heightened reliance on semantic plausibility akin to human inference. These findings demonstrate how resource limitations drive language models toward commonsense-based reasoning strategies, offering novel insights into their underlying cognitive mechanisms.

Abstract

Language models (LMs) behave more like humans when their cognitive resources are restricted, particularly in predicting sentence processing costs such as reading times. However, it remains unclear whether such constraints similarly affect sentence comprehension strategies. Besides, existing methods do not directly target the balance between memory storage and sentence processing, which is central to human working memory. To address this issue, we propose a dual-task paradigm that combines an arithmetic computation task with a sentence comprehension task, such as "The 2 cocktail + blended 3 =..." Our experiments show that under dual-task conditions, GPT-4o, o3-mini, and o4-mini shift toward plausibility-based comprehension, mirroring humans' rational inference. Specifically, these models show a greater accuracy gap between plausible sentences (e.g., "The cocktail was blended by the bartender") and implausible sentences (e.g., "The bartender was blended by the cocktail") in the dual-task condition compared to the single-task conditions. These findings suggest that constraints on the balance between memory and processing resources promote rational inference in LMs. More broadly, they support the view that human-like sentence comprehension fundamentally arises from the allocation of limited cognitive resources.

Problem

• sentence comprehension
• cognitive resources
• working memory
• language models
• dual-task paradigm

Innovation

• dual-task paradigm
• sentence comprehension
• plausibility-based inference
• cognitive resource constraints
• working memory