Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning

2026-09-16

AI Summary

本文提出一种新的大语言模型解码框架ME-Decoding，通过马氏距离驱动的目标优化候选词选择，解决现有方法忽略语义关系和计算复杂的问题。

Abstract

We introduce Mahalanobis-Ensemble Decoding (ME-Decoding), a novel Large Language Model (LLM) decoding framework that frames candidate token selection as ensemble pruning. Existing selection strategies rely predominantly on scalar probabilities, ignoring geometric semantic relationships and causing candidate redundancy. Meanwhile, current geometry-aware methods often require complex optimization or directly reweighting the original token probabilities, leading to significant computational overhead or inference instability. To address this, we formulate decoding as a subset optimization problem using a Mahalanobis distance-driven objective to enhance semantic diversity while preserving high probabilities. Specifically, we dynamically discount redundant generation paths using a token similarity matrix, constructed via an adaptive-bandwidth kernel over token embeddings. We further devise an efficient greedy selection algorithm with near-linear complexity in the candidate size under early stopping, while establishing its theoretical approximation guarantees. This renders ME-Decoding a robust, plug-and-play module with negligible inference overhead. Extensive experiments across diverse reasoning and generation tasks demonstrate that our method consistently achieves strong performance.

Problem

• Large Language Model
• Decoding
• Semantic Diversity
• Computational Overhead
• Token Redundancy

Innovation

• Mahalanobis-Ensemble Decoding
• ensemble pruning
• semantic diversity
• greedy selection algorithm
• adaptive-bandwidth kernel