Entropy-informed Decoding: Adaptive Information-Driven Branching

2026-05-10

AI Summary

Existing decoding strategies for large language models struggle to balance output quality and computational efficiency: sampling often gets trapped in local optima, while fixed-beam search incurs redundant computation. This work proposes EDEN, a plug-and-play, model-agnostic adaptive decoding framework that, for the first time, formulates decoding as a noise maximization problem. EDEN dynamically adjusts the branching factor based on the entropy of the output distribution—expanding to multiple candidates under high entropy and converging toward greedy selection under low entropy. Theoretically, this entropy-monotonic strategy is shown to outperform fixed-branch approaches under identical computational budgets, with an explicit regret bound provided. Experiments demonstrate that EDEN significantly improves accuracy across diverse tasks such as mathematical reasoning, code generation, and scientific question answering, achieving a superior trade-off between precision and computational cost.

Abstract

Large language models (LLMs) achieve remarkable generative performance, yet their output quality is dependent on the decoding strategy. While sampling-based methods (e.g., top-k, nucleus) and search-and-select based methods (e.g., beam search, best-of-n, majority voting) can improve upon greedy decoding, both approaches suffer from limitations: sampling generally commits to a single path, while search often expends excessive computation regardless of task complexity. To address these, we introduce Entropy-informed decoding (EDEN), a plug-and-play, model-agnostic decoding framework that adaptively allocates computation based on the model's own uncertainty, approximating higher-width beam search with fewer expansions. At each generation step, EDEN estimates the entropy of the output token distribution and adjusts the branching factor monotonically with the entropy, expanding more candidates in high-entropy regions and following a greedier path in low-entropy regions, improving token efficiency. Experiments across complex tasks, including mathematical reasoning, code generation, and scientific questions, demonstrate that EDEN consistently improves output quality over existing decoding strategies, achieving better accuracy-expansion trade-offs than fixed-width beam search. By treating next-token selection as a noisy maximisation problem, we prove that branching factors monotone in entropy are guaranteed to find better (i.e. more probable) continuations than any fixed branching factor within the same total expansion budget, and derive explicit regret rates characterising the benefit of the adaptive allocation.

Problem

• decoding strategy
• large language models
• computational efficiency
• output quality
• entropy

Innovation

• entropy-informed decoding
• adaptive branching
• token efficiency
• uncertainty-aware generation
• decoding strategy