CODED-SMOOTHING: Coding Theory Helps Generalization

2025-09-30

AI Summary

This work addresses two critical challenges in deep learning: poor generalization and insufficient adversarial robustness. To this end, we propose Coded-Smoothing—a novel module that integrates coding-theoretic principles into machine learning. During training, it applies a learnable linear transformation to inputs, inducing implicit regularization; during inference, it introduces stochastic linear coding to amplify model uncertainty and thereby enhance robustness against gradient-based attacks (e.g., PGD, FGSM). This is the first systematic application of coding computation theory to both regularization and inference-time randomization, supporting both supervised and unsupervised learning with negligible computational overhead. Extensive experiments demonstrate significant improvements in generalization across diverse tasks—including image classification and self-supervised learning—and achieve state-of-the-art robustness on standard adversarial benchmarks.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Applications of cryptography

Abstract

We introduce the coded-smoothing module, which can be seamlessly integrated into standard training pipelines, both supervised and unsupervised, to regularize learning and improve generalization with minimal computational overhead. In addition, it can be incorporated into the inference pipeline to randomize the model and enhance robustness against adversarial perturbations. The design of coded-smoothing is inspired by general coded computing, a paradigm originally developed to mitigate straggler and adversarial failures in distributed computing by processing linear combinations of the data rather than the raw inputs. Building on this principle, we adapt coded computing to machine learning by designing an efficient and effective regularization mechanism that encourages smoother representations and more generalizable solutions. Extensive experiments on both supervised and unsupervised tasks demonstrate that coded-smoothing consistently improves generalization and achieves state-of-the-art robustness against gradient-based adversarial attacks.

Problem

• Improving generalization in supervised and unsupervised learning
• Enhancing model robustness against adversarial perturbations
• Adapting coded computing principles for regularization mechanisms

Innovation

• Integrates coded-smoothing module into training pipelines
• Uses linear data combinations for regularization and robustness
• Encourages smoother representations to improve generalization