Inferring from Logits: Exploring Best Practices for Decoding-Free Generative Candidate Selection

2025-01-28

AI Summary

This study addresses the inefficiency and training instability of large language models (LLMs) in multi-token candidate selection tasks—such as preference ranking, multiple-choice question answering, and clinical decision-making—arising from reliance on autoregressive decoding. We propose a decoding-free direct candidate selection paradigm. Through systematic evaluation of over ten logits-based probability estimation methods—including logsumexp, softmax truncation, and token-level weighted aggregation—we identify critical principles governing logits reweighting, normalization, and architecture-aware adaptation, and formulate task- and model-informed selection strategies. Evaluated across five multiple-choice QA benchmarks and four large-scale clinical decision benchmarks (with candidate sets up to >100K), our approach demonstrates consistent accuracy gains and improved gradient differentiability across LLaMA, Qwen, and Phi families. This work provides the first comprehensive, empirically grounded guide for efficient, trainable non-autoregressive generation in candidate selection.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Generative Language Models rely on autoregressive decoding to produce the output sequence token by token. Many tasks such as preference optimization, require the model to produce task-level output consisting of multiple tokens directly by selecting candidates from a pool as predictions. Determining a task-level prediction from candidates using the ordinary token-level decoding mechanism is constrained by time-consuming decoding and interrupted gradients by discrete token selection. Existing works have been using decoding-free candidate selection methods to obtain candidate probability from initial output logits over vocabulary. Though these estimation methods are widely used, they are not systematically evaluated, especially on end tasks. We introduce an evaluation of a comprehensive collection of decoding-free candidate selection approaches on a comprehensive set of tasks, including five multiple-choice QA tasks with a small candidate pool and four clinical decision tasks with a massive amount of candidates, some with 10k+ options. We evaluate the estimation methods paired with a wide spectrum of foundation LMs covering different architectures, sizes and training paradigms. The results and insights from our analysis inform the future model design.

Problem

• Multi-token Selection
• Efficient Decoding
• Task-specific Applications

Innovation

• Logits-based Selection
• Medical Decision-making
• Model-agnostic Performance