LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

2026-08-13

AI Summary

This work addresses the challenge that existing language models, trained on heterogeneous data, lack clearly defined knowledge boundaries, hindering systematic investigation into how they acquire and utilize knowledge. To overcome this limitation, the authors construct LITTLECURRICULUM—a meticulously curated 88-billion-token corpus derived exclusively from U.S. K–5 educational standards—and train from scratch a 5-billion-parameter model, LittleLearner, which serves as a controllable and interpretable research sandbox with well-demarcated knowledge limits. Experimental results demonstrate that LittleLearner’s linguistic capabilities are strictly bounded by the curriculum scope; while post-training and in-context learning enhance the efficiency of leveraging existing knowledge, they do not enable generalization beyond the prescribed content. This validates the framework’s efficacy for probing the mechanisms underlying knowledge acquisition and deployment in language models.

Abstract

Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LITTLECURRICULUM yields LITTLELEARNER, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LITTLECURRICULUM and LITTLELEARNER as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LITTLELEARNER better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.

Problem

• language models
• knowledge acquisition
• controlled training data
• curriculum-based learning
• developmental constraints

Innovation

• controlled pretraining
• curriculum-aligned language model
• knowledge boundaries
• developmentally restricted sandbox
• in-context learning