Nothing from Something: Can a Language Model Discover 0?

2026-06-15

AI Summary

This study investigates whether language models can autonomously discover the fundamental mathematical concept of “zero” despite its absence from their training data, thereby probing their capacity for mathematical generalization beyond the training distribution. Using GPT-2–scale models, the authors combine standard language pretraining with few-shot fine-tuning and evaluate performance through zero-shot and few-shot experimental settings. The results demonstrate that models without explicit exposure to zero cannot infer it; however, fine-tuning with only dozens to hundreds of examples containing zero substantially improves performance. Notably, language pretraining reduces the required number of such examples by approximately 50%, providing the first systematic evidence that linguistic competence facilitates the acquisition of foundational mathematical concepts.

Abstract

AI systems based on artificial neural networks are being developed with aspirations of pushing the boundary of human mathematical knowledge. A key question for these systems is how much they can reach beyond their training data. Mathematical discovery requires a strong form of out of distribution generalization; the ability to hypothesize genuinely new - and potentially logically more powerful - mathematical structures. It has been hypothesized that language abilities support such generalizations in human cognition. In this work, we use simple arithmetic as a case study for examining how modern AI models could expand their mathematical horizons, evaluating whether these models can independently discover the concept of "zero". We show that We show that (1) language models of a GPT-2 size are unable to perform this generalization at test time regardless of language pretraining, but (2) models can improve substantially after training on tens or hundreds of examples of zero. Additionally, we find that language pretraining reduces the number of required examples by approximately $50\%$, showing that language abilities can scaffold mathematical discovery in neural models.

Problem

• mathematical discovery
• zero
• out-of-distribution generalization
• language models
• neural networks

Innovation

• mathematical discovery
• out-of-distribution generalization
• language pretraining
• concept of zero
• neural scaffolding