Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance
This work investigates the minimal labeled data requirements for specialized small-scale models to surpass general-purpose large language models (LLMs) under few-shot settings (10–1,000 samples). Using eight text classification tasks, we systematically evaluate performance inflection points of seven models—including LLaMA and BERT—across fine-tuning, instruction tuning, prompt engineering, and in-context learning. We find that, on average, multi-class tasks require ≤100 samples for small models to match or exceed LLM performance, whereas binary classification may demand up to 5,000 samples. We propose a robust sample-requirement estimation framework that explicitly accounts for performance variance, improving estimation accuracy by 100–200% (up to 1,500% in extreme cases). Our core contribution is the quantitative characterization of how task properties—particularly class count and output variance—affect small-model data efficiency. This yields reproducible, task-aware guidelines for determining optimal annotation budgets in low-resource NLP scenarios.