When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation

2026-06-19

AI Summary

This work addresses the high training and inference costs in Chain-of-Thought (CoT) distillation caused by verbose reasoning traces from teacher models, noting that existing compression approaches lack systematic disentanglement of key factors. The study introduces the first structured decomposition of CoT compression into three dimensions: importance criteria, reconstruction granularity, and compression budget, integrating selective pruning with generative rewriting. Comprehensive experiments across mathematical and general domains, as well as short and long CoT settings, reveal that step-level importance criteria converge toward a shared reasoning backbone; mathematical tasks are sensitive to structural perturbations, whereas general tasks benefit from aggressive rewriting; and compression during training does not necessarily reduce inference cost. The findings yield condition-aware compression guidelines that elucidate non-trivial interactions among granularity, domain specificity, and computational cost.

Abstract

Chain-of-Thought (CoT) distillation transfers multi-step reasoning from large reasoning models to smaller students, but verbose teacher traces inflate both training and inference cost. Existing CoT compression methods fall into two families, selective pruning and generative rewriting, yet prior studies have left key factors entangled: granularity is confounded with importance criteria in pruning, restructuring level is rarely isolated in rewriting, and compression budgets are not systematically evaluated across domains or regimes. We recast CoT compression along three dimensions: importance criterion, restructuring level, and compression budget. Sweeping these across two model families, Math and General domains, and Long-/Short-CoT regimes, we find that (i) importance criterion utility is strictly governed by granularity: step-level criteria converge on a shared reasoning backbone, while token-level pruning requires symbol-aware signals to preserve the logical core; (ii) restructuring level inverts across domains: Math degrades monotonically with structural disruption, while aggressive rewriting acts as a denoiser on General tasks; (iii) training-time compression does not necessarily translate to inference-time savings: Long-CoT students retain verbose habits despite concise supervision, making the training ratio an optimistic lower bound on deployment cost. These findings yield condition-aware guidelines for matching compression to deployment context.

Problem

• Chain-of-Thought compression
• importance criterion
• restructuring level
• compression budget
• distillation

Innovation

• Chain-of-Thought distillation
• CoT compression
• condition-aware analysis
• restructuring level
• compression budget