🤖 AI Summary
This study investigates how the syntactic arrangement of test code—whether co-located with or separated from implementation code—affects the quality of code generated by large language models. Through a large-scale empirical analysis, it compares inline tests (e.g., Python doctests) against separated tests (e.g., Rust #[test] attributes), evaluating model outputs using the SEGA framework across three dimensions: determinism, retention, and correctness. The work reveals, for the first time, that co-locating tests consistently and significantly enhances code quality across twelve prominent models, including six Transformer variants and one gated linear RNN. Results show inline tests achieve near-perfect retention rates (≈100%) and high correctness (92–100%), whereas separated tests not only expose performance disparities among models but also decouple correctness from retention, offering critical insights for future model design.
📝 Abstract
AI coding assistants increasingly generate code alongside tests. How developers structure test code, whether inline with the implementation or in separate blocks, has traditionally been a matter of testing philosophy. We investigate whether this choice affects AI code generation quality.
We conduct a large-scale empirical study (830+ generated files, 12 models, 3 providers) using SEGA, a three-dimensional evaluation framework measuring Determinism, Preservation, and Correctness. Comparing inline test syntax (Python doctests) against separated test syntax (Rust #[test] blocks) on a d-ary heap implementation, we find that: (1) inline tests yield near-perfect preservation (100%) and correctness (92-100%) across all models; (2) separated tests expose stark model-tier gaps (0-100% correctness) and independence between preservation and correctness; (3) model behavior evolves across generations, and notably one model breaks the test suppression pattern of its three predecessors; (4) mechanistic analysis on 7 open-source architectures (6 transformers and a gated-linear Recurrent Neural Network (RNN)) reveals inline test markers receive 2.8-4.4$\times$ stronger attention in 5/7 models, with causal validation via knockout and steering experiments on the 4 code-specialized transformers and RWKV-6; the co-location mechanism extends to a non-transformer architecture, suggesting the design recommendation is robust to future architectural shifts. In the Foundation Model era, test syntax structure is a software design concern: co-locating tests with implementation code produces measurably better AI-generated code. This arxiv long version includes appendices that further qualify the effect as bounded by both model capability and programming language.