Tiny language models

2025-07-20

AI Summary

The prohibitively high computational and data requirements for pretraining large language models (LLMs) hinder broad research participation. Method: This paper systematically investigates whether tiny language models (TLMs)—e.g., BERT-1 and BERT-6 with only 1–6 layers—can reproduce key qualitative capabilities of LLMs. We conduct lightweight pretraining on a Wikipedia subset and evaluate on FewRel, AGNews, and DBPedia. Contribution/Results: (1) Pretraining consistently improves downstream performance even in ultra-shallow architectures; (2) soft ensembling of multiple TLMs achieves classification accuracy comparable to deeper models while enabling low-latency inference; (3) performance gains correlate positively with pretraining data volume and token overlap between pretraining and downstream tasks. This work provides the first empirical validation that effective language modeling is feasible with TLMs under small-scale, low-resource conditions, opening a new pathway toward democratized NLP research.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

A prominent achievement of natural language processing (NLP) is its ability to understand and generate meaningful human language. This capability relies on complex feedforward transformer block architectures pre-trained on large language models (LLMs). However, LLM pre-training is currently feasible only for a few dominant companies due to the immense computational resources required, limiting broader research participation. This creates a critical need for more accessible alternatives. In this study, we explore whether tiny language models (TLMs) exhibit the same key qualitative features of LLMs. We demonstrate that TLMs exhibit a clear performance gap between pre-trained and non-pre-trained models across classification tasks, indicating the effectiveness of pre-training, even at a tiny scale. The performance gap increases with the size of the pre-training dataset and with greater overlap between tokens in the pre-training and classification datasets. Furthermore, the classification accuracy achieved by a pre-trained deep TLM architecture can be replicated through a soft committee of multiple, independently pre-trained shallow architectures, enabling low-latency TLMs without affecting classification accuracy. Our results are based on pre-training BERT-6 and variants of BERT-1 on subsets of the Wikipedia dataset and evaluating their performance on FewRel, AGNews, and DBPedia classification tasks. Future research on TLM is expected to further illuminate the mechanisms underlying NLP, especially given that its biologically inspired models suggest that TLMs may be sufficient for children or adolescents to develop language.

Problem

• Exploring if tiny language models mimic key features of large models
• Assessing performance gap between pre-trained and non-pre-trained tiny models
• Investigating scalable low-latency alternatives to resource-intensive large models

Innovation

• Tiny language models mimic LLM qualitative features
• Pre-training effectiveness shown in tiny-scale models
• Soft committee enables low-latency TLMs accurately