Scaling Laws for Task-Specific LLM Distillation

2026-06-23

AI Summary

This work addresses the challenge of deploying large language models in quantitative finance, where latency and cost constraints necessitate efficient compression methods that preserve both task performance and computational efficiency. The authors propose a KL divergence–based distillation framework enhanced with chain-of-thought supervision, integrating logit distillation, LoRA fine-tuning, and iterative structured pruning, along with a novel supervised loss function. By constructing the FinHeadlineMix dataset, they systematically investigate the effects of data scale, compression ratio, supervision format, and pruning strategy. Their analysis reveals that while task-specific performance degrades predictably, general-purpose capabilities collapse prematurely. Building on these findings, the study establishes predictable scaling laws for both domain-specific and general knowledge under compression, offering practical guidance for informed model compression decisions in real-world applications.

Abstract

Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives empirical scaling laws for domain-specific LLM compression, quantifying how in-domain and general knowledge performance scale with dataset size, compression ratio, supervision format, and iterative pruning schedule. Using quantitative finance as our application domain, we compare logit-based and LoRA-based distillation under iterative structural pruning, introducing a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces. In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point; supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases. We release the headline dataset FinHeadlineMix, scaling law results, and practical recommendations to provide a reusable framework for domain-specific compression decisions.

Problem

• LLM compression
• domain-specific distillation
• scaling laws
• knowledge retention
• model deployment

Innovation

• scaling laws
• task-specific distillation
• chain-of-thought supervision
• iterative pruning
• LoRA-based distillation