The Binary and Ternary Quantization Can Improve Feature Discrimination

2025-04-18

AI Summary

Conventional quantization research assumes that quantization error directly predicts classification performance—a hypothesis lacking theoretical justification and contradicted by empirical evidence (e.g., binary/tri-level quantization often incurs large reconstruction error yet preserves or even improves accuracy). Method: This work departs from the error-minimization paradigm and instead directly evaluates the discriminative power of quantized features. Contribution/Results: We theoretically and empirically establish that {0,1} binary and {0,±1} ternary quantization can enhance—rather than degrade—feature separability, challenging the prevailing “lower error implies higher accuracy” assumption. Extensive experiments across multimodal data (images, speech, text) and multiple benchmark datasets demonstrate that our low-bit quantization methods achieve classification accuracy on par with or superior to full-precision models. These findings reveal that the fundamental impact of quantization on classification lies in discriminability reconfiguration—not reconstruction fidelity optimization.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Computer Vision: Representation Learning for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In machine learning, quantization is widely used to simplify data representation and facilitate algorithm deployment on hardware. Given the fundamental role of classification in machine learning, it is crucial to investigate the impact of quantization on classification. Current research primarily focuses on quantization errors, operating under the premise that higher quantization errors generally result in lower classification performance. However, this premise lacks a solid theoretical foundation and often contradicts empirical findings. For instance, certain extremely low bit-width quantization methods, such as ${0,1}$-binary quantization and ${0, pm1}$-ternary quantization, can achieve comparable or even superior classification accuracy compared to the original non-quantized data, despite exhibiting high quantization errors. To more accurately evaluate classification performance, we propose to directly investigate the feature discrimination of quantized data, instead of analyzing its quantization error. Interestingly, it is found that both binary and ternary quantization methods can improve, rather than degrade, the feature discrimination of the original data. This remarkable performance is validated through classification experiments across various data types, including images, speech, and texts.

Problem

• Investigates impact of quantization on classification performance
• Challenges premise linking high quantization errors to low accuracy
• Shows binary/ternary quantization improves feature discrimination

Innovation

• Binary and ternary quantization enhance feature discrimination
• Directly evaluate feature discrimination over quantization errors
• Validated across images, speech, and text data