DenseSteer: Steering Small Language Models towards Dense Math Reasoning

2026-05-27

AI Summary

Small-scale language models (≤3B parameters) exhibit significantly weaker performance than larger counterparts on multi-step mathematical reasoning tasks. This work identifies and leverages a previously unexplored characteristic termed “dense reasoning”—referring to high-information-density, low-step-count reasoning paths—and introduces a training-free, inference-time steering framework that guides small models toward such reasoning modes by modulating their internal representations. The proposed method incurs no additional computational overhead or token-level negative log-likelihood penalty, yet consistently improves accuracy across the Qwen-2.5 model family and multiple mathematical reasoning benchmarks. These results demonstrate the effectiveness and generality of the proposed training-agnostic steering mechanism in enhancing the reasoning capabilities of compact language models.

Abstract

Large language models (LLMs) demonstrate strong chain-of-thought (CoT) reasoning abilities, while smaller models (<= 3B parameters) significantly underperform on multi-step reasoning tasks. Based on empirical analyses of the Qwen-2.5 model family on math reasoning benchmarks, we find that more proficient reasoning is associated with fewer reasoning steps but higher information density per step, a property we term Dense Reasoning. Motivated by this observation, we propose DenseSteer, a training-free inference-time steering framework that enhances small-model reasoning by modulating internal representations toward dense reasoning patterns. Experiments show that our method yields consistent accuracy improvements without increasing token-level Negative Log-Likelihood, highlighting dense reasoning as an effective structural approach to mathematical problem solving.

Problem

• small language models
• math reasoning
• chain-of-thought
• dense reasoning
• multi-step reasoning

Innovation

• Dense Reasoning
• Small Language Models
• Inference-time Steering
• Mathematical Reasoning
• Chain-of-Thought