When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

2025-12-21

AI Summary

Catastrophic forgetting remains a critical challenge in continual learning (CL) for large language models (LLMs), particularly under resource constraints. Method: This work systematically investigates the synergistic effects of quantization precision (FP16, INT8, INT4) and minimal replay buffer strategies across multi-task CL sequences spanning natural language understanding (NLU), mathematical reasoning, and code generation. Contribution/Results: Contrary to the common assumption that higher precision yields better performance, INT8 quantization achieves the optimal trade-off between knowledge retention and adaptation to new tasks. We identify, for the first time, that quantization noise serves as an implicit regularizer mitigating forgetting. Remarkably, only 0.1%–2% replay buffer suffices for substantial CL gains. Key results: INT8 models achieve 8–15% higher forward accuracy on final tasks versus FP16; in code generation, INT4 attains 40% accuracy—double FP16’s 20%. This work delivers dual breakthroughs in inference efficiency and continual learning capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency. We systematically investigate the interplay between quantization precision (FP16, INT8, INT4) and replay buffer strategies in large language models, revealing unexpected dynamics. While FP16 achieves superior initial task performance (74.44% on NLU), we observe a striking inversion on subsequent tasks: quantized models outperform FP16 by 8-15% on final task forward accuracy, with INT4 achieving nearly double FP16's performance on Code generation (40% vs 20%). Critically, even minimal replay buffers (0.1%) dramatically improve retention - increasing NLU retention after Math training from 45% to 65% across all precision levels - with INT8 consistently achieving the optimal balance between learning plasticity and knowledge retention. We hypothesize that quantization-induced noise acts as implicit regularization, preventing the overfitting to new task gradients that plagues high-precision models. These findings challenge the conventional wisdom that higher precision is always preferable, suggesting instead that INT8 quantization offers both computational efficiency and superior continual learning dynamics. Our results provide practical guidelines for deploying compressed models in continual learning scenarios: small replay buffers (1-2%) suffice for NLU tasks, while Math and Code benefit from moderate buffers (5-10%), with quantized models requiring less replay than FP16 to achieve comparable retention. Code is available at https://github.com/Festyve/LessIsMore.

Problem

• Investigates how quantization precision affects continual learning in large language models.
• Examines the interplay between quantization and replay buffers for knowledge retention.
• Challenges the assumption that higher precision is always better for continual learning.

Innovation

• INT8 quantization balances learning plasticity and retention
• Quantization noise acts as implicit regularization to prevent overfitting
• Small replay buffers improve retention across all precision levels