Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity

2026-04-22

AI Summary

This work addresses the challenge of learning robust linguistic representations under data-scarce conditions, where standard Transformers often struggle. Inspired by cognitive science, the study introduces working memory constraints as an inductive bias into the Transformer architecture, proposing a novel attention module that combines a fixed-width context window with a temporally decaying mechanism. Building upon a modified GPT-2 architecture, the model is trained from scratch on limited corpora. Experimental results demonstrate significant improvements in grammaticality judgment accuracy—assessed via the BLiMP benchmark—and reveal a strong alignment between the model’s predictions and human reading times. These findings substantiate the efficacy and cognitive plausibility of incorporating working memory-inspired mechanisms into low-resource language modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We investigate the integration of human-like working memory constraints into the Transformer architecture and implement several cognitively inspired attention variants, including fixed-width windows based and temporal decay based attention mechanisms. Our modified GPT-2 models are trained from scratch on developmentally plausible datasets (10M and 100M words). Performance is evaluated on grammatical judgment tasks (BLiMP) and alignment with human reading time data. Our results indicate that these cognitively-inspired constraints, particularly fixed-width attention, can significantly improve grammatical accuracy especially when training data is scarce. These constrained models also tend to show a stronger alignment with human processing metrics. The findings suggest that such constraints may serve as a beneficial inductive bias, guiding models towards more robust linguistic representations, especially in data-limited settings.

Problem

• working memory
• data scarcity
• Transformer
• inductive bias
• language acquisition

Innovation

• working memory constraints
• cognitively-inspired attention
• fixed-width attention
• data scarcity
• inductive bias