Select to Think: Unlocking SLM Potential with Local Sufficiency

2026-04-29

AI Summary

This work addresses the limited reasoning capabilities of small language models (SLMs) and the high cost or inefficacy of existing enhancement methods that rely on large language model (LLM) calls or struggle to distill complex generative distributions. The authors propose the SELECT TO THINK (S2T) framework, grounded in the “local sufficiency” hypothesis—that tokens preferred by LLMs for reasoning are typically within the top-K predictions of SLMs. Rather than generating outputs, the LLM acts as a selector over SLM candidates, and this selection logic is distilled into the SLM via S2T-LOCAL, enabling autonomous reranking without external intervention. Experiments show that a 1.5B-parameter SLM covers 95% of a 32B LLM’s token selections within its top-8 candidates, and S2T-LOCAL improves greedy decoding performance by 24.1% on average—matching the accuracy of 8-path self-consistency while requiring only single-trajectory computation.

Abstract

Small language models (SLMs) offer computational efficiency for scalable deployment, yet they often fall short of the reasoning power exhibited by their larger counterparts (LLMs). To mitigate this gap, current approaches invoke an LLM to generate tokens at points of reasoning divergence, but these external calls introduce substantial latency and costs. Alternatively, standard distillation is often hindered by the capacity limitation, as SLMs struggle to accurately mimic the LLM's complex generative distribution. We address this dilemma by identifying local sufficiency: at divergence points, the LLM's preferred token consistently resides within the SLM's top-K next-token predictions, even when failing to emerge as the SLM top-1 choice. We therefore propose SELECT TO THINK (S2T), which reframes the LLM's role from open-ended generation to selection among the SLM's proposals, simplifying the supervision signal to discrete candidate rankings. Leveraging this, we introduce S2T-LOCAL, which distills the selection logic into the SLM, empowering it to perform autonomous re-ranking without inference-time LLM dependency. Empirically, we demonstrate that a 1.5B SLM's top-8 candidates capture the 32B LLM's choice with 95% hit rate. Translating this potential into performance, S2T-LOCAL improves greedy decoding by 24.1% on average across benchmarks, effectively matching the efficacy of 8-path self-consistency while operating with single-trajectory efficiency.

Problem

• small language models
• reasoning capability
• distillation
• latency
• computational efficiency

Innovation

• local sufficiency
• small language models
• distillation
• token selection
• autonomous reranking