MCEL: Margin-Based Cross-Entropy Loss for Error-Tolerant Quantized Neural Networks

2026-03-05

AI Summary

This work addresses the sensitivity of quantized neural networks to bit errors in approximate computing and unreliable memory systems by proposing an efficient, training-free fault-tolerant approach. By establishing, for the first time, a theoretical connection between classification margins and robustness to bit errors, the authors introduce tunable and interpretable margin parameters and a plug-and-play Margin Cross-Entropy Loss (MCEL). This loss function explicitly enhances logit-level separation while preserving the optimization properties of standard cross-entropy. Extensive experiments demonstrate that MCEL consistently improves fault tolerance across diverse network architectures, quantization schemes, and datasets, achieving up to a 15% accuracy gain under a 1% bit error rate—significantly outperforming conventional error-injection training methods.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Robustness to bit errors is a key requirement for the reliable use of neural networks (NNs) on emerging approximate computing platforms and error-prone memory technologies. A common approach to achieve bit error tolerance in NNs is injecting bit flips during training according to a predefined error model. While effective in certain scenarios, training-time bit flip injection introduces substantial computational overhead, often degrades inference accuracy at high error rates, and scales poorly for larger NN architectures. These limitations make error injection an increasingly impractical solution for ensuring robustness on future approximate computing platforms and error-prone memory technologies. In this work, we investigate the mechanisms that enable NNs to tolerate bit errors without relying on error-aware training. We establish a direct connection between bit error tolerance and classification margins at the output layer. Building on this insight, we propose a novel loss function, the Margin Cross-Entropy Loss (MCEL), which explicitly promotes logit-level margin separation while preserving the favorable optimization properties of the standard cross-entropy loss. Furthermore, MCEL introduces an interpretable margin parameter that allows robustness to be tuned in a principled manner. Extensive experimental evaluations across multiple datasets of varying complexity, diverse NN architectures, and a range of quantization schemes demonstrate that MCEL substantially improves bit error tolerance, up to 15 % in accuracy for an error rate of 1 %. Our proposed MCEL method is simple to implement, efficient, and can be integrated as a drop-in replacement for standard CEL. It provides a scalable and principled alternative to training-time bit flip injection, offering new insights into the origins of NN robustness and enabling more efficient deployment on approximate computing and memory systems.

Problem

• bit error tolerance
• quantized neural networks
• approximate computing
• error-prone memory
• robustness

Innovation

• Margin-based Cross-Entropy Loss
• bit error tolerance
• quantized neural networks
• error-prone memory
• approximate computing