Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens

2026-04-29

AI Summary

This work addresses the high computational cost of large language model inference and the underutilization of token-level trajectory structure. The authors propose a model-agnostic inference compression method that classifies tokens into structural (low-entropy) and organic (high-entropy) categories via entropy analysis, then applies cross-token BPE merging along the reasoning trajectory to form supertokens. These compressed units are reinforced through supervised fine-tuning to guide the model’s generation process. Evaluated across three model families and five mathematical reasoning benchmarks, the approach reduces average inference length by 8.1% without significant accuracy degradation. Moreover, supertokens serve as interpretable labels for reasoning actions, revealing systematic differences between correct and erroneous reasoning pathways.

Abstract

Reasoning in Large Language Models incurs significant inference-time compute, yet the token-level information structure of reasoning traces remains underexplored. We observe that reasoning tokens split into two functional types: low-entropy \textit{structural} tokens (recurring phrases that scaffold the reasoning process) and higher-entropy \textit{organic} tokens (problem-specific content that drives toward a solution). This asymmetry motivates a simple, model-agnostic compression pipeline: apply cross-word BPE merges on a model's own reasoning traces to derive \textit{supertokens} that capture frequent structural patterns, then teach the model to adopt them via supervised fine-tuning. Across three model families and five mathematical reasoning benchmarks, our approach shortens reasoning traces by 8.1\% on average with no statistically significant accuracy loss on any model--benchmark pair. Beyond compression, supertokens act as interpretable reasoning-move annotations (backtracking, verification, strategy shifts), exposing the model's high-level strategy at a glance. Analyzing transitions between structural categories reveals systematic differences between correct and incorrect traces: correct traces show productive recovery (backtracking followed by strategy shifts and verification), while incorrect traces are dominated by confusion cycles (repeated hedging and unresolved contradictions). These diagnostic signals suggest applications in reward shaping and early stopping for RL-based reasoning training.

Problem

• LLM reasoning
• inference compression
• reasoning traces
• token-level structure
• computational efficiency

Innovation

• supertokens
• entropy-guided compression
• reasoning trace analysis
• interpretable reasoning
• model-agnostic compression