Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning

2025-08-27

AI Summary

This work addresses the challenge of designing effective curriculum learning strategies for language model pretraining by leveraging human intuition about linguistic difficulty. We propose a label-driven curriculum framework grounded in human-annotated article-level “simplicity” labels from Simple Wikipedia, and systematically evaluate it on BERT-tiny. Unlike heuristic-based approaches (e.g., sentence length, word frequency) or dynamic, model-dependent curricula, our method achieves significant and consistent reductions in both overall and simplicity-specific perplexity—particularly improving the model’s ability to capture simple language structures. The key contribution is the first empirical demonstration that human linguistic intuition serves as a reliable, low-cost, and high-yield signal for curriculum design, offering an interpretable, easily deployable paradigm for curriculum learning in language modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Curriculum learning (CL) aims to improve training by presenting data from "easy" to "hard", yet defining and measuring linguistic difficulty remains an open challenge. We investigate whether human-curated simple language can serve as an effective signal for CL. Using the article-level labels from the Simple Wikipedia corpus, we compare label-based curricula to competence-based strategies relying on shallow heuristics. Our experiments with a BERT-tiny model show that adding simple data alone yields no clear benefit. However, structuring it via a curriculum -- especially when introduced first -- consistently improves perplexity, particularly on simple language. In contrast, competence-based curricula lead to no consistent gains over random ordering, probably because they fail to effectively separate the two classes. Our results suggest that human intuition about linguistic difficulty can guide CL for language model pre-training.

Problem

• Defining and measuring linguistic difficulty in curriculum learning
• Investigating human-curated simple language as signal for curriculum
• Comparing label-based curricula with shallow heuristic strategies

Innovation

• Human-curated simple language for curriculum learning
• Label-based curricula outperform shallow heuristics
• Human intuition guides effective linguistic difficulty ordering