On Language Generation in the Limit with Bounded Memory

2026-05-28

AI Summary

This study addresses the problem of incrementally generating valid samples from an unknown target language under finite memory constraints, characterizing how such limitations affect both language generation and recognition capabilities. It extends bounded-memory learning theory—previously studied primarily in classification—to language generation for the first time, leveraging frameworks from inductive inference, combinatorial tools (including Sperner’s theorem and symmetric chain decomposition), and density analysis. The work establishes that any countably infinite language can be generated without memory under mild conditions, derives the optimal minimax generation density for finite language classes, and proves that adaptive storage strictly outperforms sliding-window memory. Furthermore, it introduces an approximate identification mechanism enabling incremental limit identification of finite language classes.

Abstract

We study language generation in the limit under bounded memory. In this task, a learner observes examples from an unknown target language one at a time and must eventually output only new valid examples. Prior work assumes access to the entire history, a strong assumption since realistic algorithms retain limited past information. Classical work in learning theory shows memory constraints dramatically alter learnability; we extend this to language generation. First, we study memoryless generators. Under a mild enumeration restriction, every countable collection of infinite languages remains generable without memory. Without this restriction, we exactly characterize when memoryless generation is possible. For finite collections, we characterize the optimal minimax density achievable by memoryless generators -- the best density guaranteed against any collection of a given size. This combinatorial bound relies on Sperner's theorem and symmetric chain decompositions. We further show that a sliding window of the last $W$ examples does not improve this worst-case density, whereas allowing it to store $b$ adaptively chosen past examples improves the achievable density for every $b \geq 1$. Finally, we revisit identification in the limit, where the learner must converge to a single correct hypothesis for the target language. We focus on its incremental variant, where the learner remembers only its previous guess. Here, although exact identification fails on a collection of just three languages, a mild relaxation requiring convergence to an ``approximate'' version of the target is achievable for every finite collection. These results show bounded memory affects these tasks differently: generation remains achievable for every countable collection, while density and identification are confined to finite collections, with guarantees weakening as the collection grows.

Problem

• language generation
• bounded memory
• learnability
• identification in the limit
• memoryless generators

Innovation

• bounded memory
• language generation in the limit
• memoryless generators
• minimax density
• incremental identification