Memorization Diagnostics for Code LLMs Should be Scale-Aware

2026-08-12

AI Summary

Existing diagnostic methods for assessing memorization in large code models struggle to disentangle memorization from representational capacity as model scale increases, leading to distorted evaluations. This work proposes a novel paradigm that decouples representational load from memorization behavior through invertible mathematical transformations, complemented by systematic analyses employing synonym obfuscation, dead code insertion, and log-probability probing techniques. The study reveals that current probing approaches significantly fail on large-scale models, while the models themselves exhibit strong robustness to diverse surface forms of code. These findings challenge the validity of relying solely on contaminated benchmarks to evaluate memorization and provide both theoretical grounding and methodological support for reassessing the generalization capabilities of large code models.

Abstract

The extent to which large language models for code rely on memorization over genuine understanding remains highly debated. While current literature frequently reports widespread memorization, evaluating the underlying probing techniques across dense architectures reveals a severe breakdown in their utility at scale. Traditional encoder-style probes using perturbations such as synonym fuzzing or dead-code insertion struggle to expose memorization in scaled models, even on known-contaminated benchmarks, and decoder-style probes that rely on log probabilities show similar performance degradation. The specific mode of failure for these probes, particularly why such techniques disrupt smaller models but fail to impact larger ones, motivates us to untangle representation load from memorization rather than treating them as a single phenomenon. By applying invertible mathematical transforms to numeric problems, we isolate these two factors and reveal that scaled encoders successfully absorb substantial representation load while still converging on the correct family of solutions. In practical software engineering, this ability to adapt to varying surface forms is what truly matters for usability and generalizability in LLM and agentic applications. Whether a specific solution was seen during training becomes a much less pressing question because although memorization inflates scores on contaminated benchmarks, factoring out representation load makes it debatable how much we should truly care if a functional answer was originally memorized. Future evaluations must therefore be built around separating these phenomena rather than relying on methodologies that quietly entangle them.

Problem

• memorization
• code LLMs
• scale-aware evaluation
• representation load
• probing techniques

Innovation

• memorization
• representation load
• scale-aware evaluation
• code LLMs
• invertible transforms