Turning Down the Heat: A Critical Analysis of Min-p Sampling in Language Models

2025-06-16

AI Summary

This work critically reproduces and evaluates Nguyen et al.’s (2024) min-p sampling method, assessing whether it improves generation quality, diversity, or their trade-off relative to standard baselines—including greedy, top-k, and top-p sampling. Method: We employ a three-pronged validation framework: rigorous human evaluation, controlled NLP benchmarks (e.g., MAUVE, Distinct-n), and LLM-as-a-Judge consistency auditing, supplemented by open-ecosystem data replication. Contribution/Results: min-p yields no statistically significant gains in quality or diversity. We identify critical flaws in the original study: misuse of statistical testing, omission of essential baselines, and inconsistent result reporting. Previously cited community evidence supporting min-p has been refuted and retracted. To our knowledge, this is the first systematic, reproducible, multi-dimensional empirical analysis exposing min-p’s limitations—establishing a new benchmark for rigorous, cross-validated evaluation of decoding strategies.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Evaluation and Analysis

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Sampling from language models impacts the quality and diversity of outputs, affecting both research and real-world applications. Recently, Nguyen et al. 2024's"Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs"introduced a new sampler called min-p, claiming it achieves superior quality and diversity over established samplers such as basic, top-k, and top-p sampling. The significance of these claims was underscored by the paper's recognition as the 18th highest-scoring submission to ICLR 2025 and selection for an Oral presentation. This paper conducts a comprehensive re-examination of the evidence supporting min-p and reaches different conclusions from the original paper's four lines of evidence. First, the original paper's human evaluations omitted data, conducted statistical tests incorrectly, and described qualitative feedback inaccurately; our reanalysis demonstrates min-p did not outperform baselines in quality, diversity, or a trade-off between quality and diversity; in response to our findings, the authors of the original paper conducted a new human evaluation using a different implementation, task, and rubric that nevertheless provides further evidence min-p does not improve over baselines. Second, comprehensively sweeping the original paper's NLP benchmarks reveals min-p does not surpass baselines when controlling for the number of hyperparameters. Third, the original paper's LLM-as-a-Judge evaluations lack methodological clarity and appear inconsistently reported. Fourth, community adoption claims (49k GitHub repositories, 1.1M GitHub stars) were found to be unsubstantiated, leading to their removal; the revised adoption claim remains misleading. We conclude that evidence presented in the original paper fails to support claims that min-p improves quality, diversity, or a trade-off between quality and diversity.

Problem

• Re-evaluates min-p sampling's claimed superiority in quality and diversity
• Challenges original human evaluations and statistical methods as flawed
• Disputes min-p's benchmark performance and community adoption claims

Innovation

• Reanalyzed human evaluations for min-p sampling
• Swept NLP benchmarks controlling hyperparameters
• Examined LLM-as-a-Judge methodological clarity