Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise

2026-05-18

AI Summary

This study investigates how over-parameterized models simultaneously memorize noisy labels and generalize to clean data under label noise rates as high as 80%. Leveraging modular arithmetic tasks and two-layer neural networks, the authors employ frequency-domain analysis and subnetwork decomposition to reveal that, despite noise fitting degrading overall generalization performance, a latent generalizing structure remains embedded within the network. They propose a task-agnostic frequency-domain method to effectively extract this hidden structure, achieving near-perfect test accuracy even under severe label noise. Additionally, they design a subnetwork partitioning strategy to disentangle memorization and generalization components; although this yields only modest performance gains, it offers a novel perspective on the dual capabilities of over-parameterized models.

Abstract

Highly over-parameterized models can simultaneously memorize noisy labels and generalize well, yet how these behaviors coexist remains poorly understood. In this work, we investigate the underlying mechanisms of this coexistence using modular arithmetic tasks under heavy label noise. Through extensive experiments on two-layer neural networks, we find that larger models tend to generalize better under appropriate optimization and model configurations, while noisy labels are memorized faster than clean data. Over-parameterized models internally form a generalization structure, but its expression in the output is suppressed by the need to fit noisy labels. Remarkably, even with 80\% label noise, near-perfect test accuracy can be achieved by extracting this internal structure using frequency-based methods. We further propose a task-agnostic method to partition networks into generalization and memorization components. Although this subnetwork improves generalization, it is limited compared with frequency-based extraction, indicating that the generalization structure is distributed across neurons and motivating the development of new tools to retrieve generalizable knowledge from over-parameterized networks.

Problem

• memorization
• generalization
• label noise
• over-parameterized models
• modular arithmetic

Innovation

• memorization-generalization coexistence
• label noise
• over-parameterized models
• frequency-based extraction
• modular arithmetic tasks