Beyond the Training Distribution: Mapping Generalization Boundaries in Neural Program Synthesis

2026-04-30

AI Summary

This work investigates whether neural program synthesis models genuinely generalize out-of-distribution or merely rely on memorization and data contamination. To this end, we construct a controlled arithmetic grammar environment, systematically enumerating millions of programs to establish an interpretable space of syntactic and semantic metrics, and rigorously partition training and test distributions. We introduce an evaluation framework distinguishing “density generalization” from “support generalization,” revealing a significant bottleneck in current models’ ability to handle syntactic novelty—evidenced by performance drops exceeding 30%. Experiments demonstrate that multi-manifold diversity sampling substantially improves generalization, and that compute scaling yields log-linear gains in out-of-distribution performance, underscoring the critical importance of diverse training data.

Abstract

Large-scale transformers achieve impressive results on program synthesis benchmarks, yet their true generalization capabilities remain obscured by data contamination and opaque training corpora. To rigorously assess whether models are truly generalizing or merely retrieving memorized templates, we introduce a strictly controlled program synthesis environment based on a domain-specific arithmetic grammar. By systematically enumerating and evaluating millions of unique programs, we construct interpretable syntactic and semantic metric spaces. This allows us to precisely map data distributions and sample train and test splits that isolate specific distributional shifts. Our experiments demonstrate that optimizing density generalization -- through diverse sampling over both semantic and syntactic spaces -- induces robust out-of-distribution generalization. Conversely, evaluating support generalization reveals that transformers severely struggle with extrapolation, experiencing a performance drop of over 30% when forced to generate syntactically novel programs. While steadily scaling up compute improves generalization, the gains follow a strictly log-linear relationship. We conclude that robust generalization requires maximizing training diversity across multiple manifolds, and our findings indicate the necessity for novel search-based approaches to break through current log-linear scaling bottlenecks.

Problem

• program synthesis
• generalization
• distribution shift
• out-of-distribution
• transformers

Innovation

• program synthesis
• out-of-distribution generalization
• syntactic-semantic metric spaces
• density generalization
• scaling laws