Four Quadrants of Difficulty: A Simple Categorisation and its Limits

2026-01-04

arXiv.org

AI Summary

This work addresses a key limitation in existing curriculum learning approaches, which often rely on task-agnostic linguistic heuristics or human intuition to estimate sample difficulty—measures that poorly reflect a model’s actual learning dynamics. The authors propose a four-quadrant difficulty classification framework that systematically analyzes difficulty signals in natural language understanding tasks along two orthogonal dimensions: human versus model assessment and task-agnostic versus task-specific characteristics. Through empirical analysis of the correlation between various difficulty metrics and model training dynamics, the study provides the first validation of the effectiveness of these four categories of difficulty signals. Findings reveal that task-agnostic features are largely independent of each other and only weakly correlated with model learning, whereas task-specific features more accurately capture the true difficulty experienced by the model. These results challenge conventional assumptions in curriculum learning and underscore the need for lightweight, task-specific difficulty estimators.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Curriculum Learning (CL) aims to improve the outcome of model training by estimating the difficulty of samples and scheduling them accordingly. In NLP, difficulty is commonly approximated using task-agnostic linguistic heuristics or human intuition, implicitly assuming that these signals correlate with what neural models find difficult to learn. We propose a four-quadrant categorisation of difficulty signals -- human vs. model and task-agnostic vs. task-dependent -- and systematically analyse their interactions on a natural language understanding dataset. We find that task-agnostic features behave largely independently and that only task-dependent features align. These findings challenge common CL intuitions and highlight the need for lightweight, task-dependent difficulty estimators that better reflect model learning behaviour.

Problem

• Curriculum Learning
• sample difficulty
• task-agnostic
• task-dependent
• neural models

Innovation

• Curriculum Learning
• Difficulty Estimation
• Task-Dependent Features
• Four-Quadrant Categorisation
• Natural Language Understanding