One-shot Entropy Minimization

2025-05-26

AI Summary

This work challenges the prevailing paradigm of large language model (LLM) post-training, which relies heavily on large-scale labeled datasets and handcrafted reward functions. We propose an entropy minimization method requiring only **a single unlabeled demonstration and 10 gradient steps**, enabling lightweight, fully unsupervised fine-tuning—without human feedback or explicit reward modeling. Our approach significantly improves output consistency and task performance across diverse benchmarks. We systematically evaluate it on 15 open and proprietary LLMs spanning 13B to 440B parameters, demonstrating competitive or superior performance to conventional RLHF and rule-based reinforcement learning on benchmarks including AlpacaEval and MT-Bench. To our knowledge, this is the first empirical demonstration that single-sample entropy minimization achieves strong generalization and optimization efficiency. The method establishes a minimalist, scalable alternative for LLM post-training. Code is publicly available.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We trained 13,440 large language models and found that entropy minimization requires only a single unlabeled data and 10 steps optimization to achieve performance improvements comparable to or even greater than those obtained using thousands of data and carefully designed rewards in rule-based reinforcement learning. This striking result may prompt a rethinking of post-training paradigms for large language models. Our code is avaliable at https://github.com/zitian-gao/one-shot-em.

Problem

• Minimizing entropy with one unlabeled data
• Achieving performance gains in 10 optimization steps
• Challenging rule-based reinforcement learning paradigms

Innovation

• Single unlabeled data for entropy minimization
• 10 steps optimization for performance improvements
• Outperforms rule-based reinforcement learning