Random Label Prediction Heads for Studying Memorization in Deep Neural Networks

2026-07-13

AI Summary

This study investigates the memorization mechanisms of deep neural networks in classification tasks and their relationship with generalization. By introducing a random label prediction head (RLP-head) at arbitrary network layers, the authors empirically estimate the Rademacher complexity of the layer’s representations based on their performance in predicting auxiliary random labels, thereby constructing a sample-level metric that quantifies memorization relative to model capacity. Leveraging this insight, they propose a novel regularization method that effectively reduces the degree of memorization. Experimental results demonstrate that the relationship between memorization and generalization is not monotonic—reducing memorization can either improve or degrade generalization performance, challenging the conventional view that overfitting is equivalent to memorization.

Abstract

We introduce a straightforward yet effective method to empirically study memorization in deep neural networks for classification tasks. Our approach augments each training sample with auxiliary random labels, which are then predicted by a random label prediction head (RLP-head). RLP-heads can be attached at arbitrary depths of a network, predicting random labels from the corresponding intermediate representation and thereby enabling analysis of how memorization capacity evolves across layers. By interpreting the RLP-head performance as an empirical estimate of Rademacher complexity, we obtain a direct measure of both sample-level memorization and model capacity. We leverage this random label accuracy metric to analyze generalization and overfitting in different models and datasets. Building on this approach, we further propose a novel regularization technique based on the output of the RLP-head, which demonstrably reduces memorization. Interestingly, our experiments reveal that reducing memorization can either improve or impair generalization, depending on the dataset and training setup. These findings challenge the traditional assumption that overfitting is equivalent to memorization and suggest new hypotheses to reconcile these seemingly contradictory results. The source code is available at https://github.com/MarlonBecker/RandomLabelHeads

Problem

• memorization
• deep neural networks
• generalization
• overfitting
• Rademacher complexity

Innovation

• random label prediction head
• memorization
• Rademacher complexity
• regularization
• generalization