Flaming-hot Initiation with Regular Execution Sampling for Large Language Models

2024-10-28

arXiv.org

AI Summary

This work addresses the low sampling efficiency and trade-off between diversity and correctness in large language models (LLMs) for reasoning tasks requiring sandbox-based verification—such as mathematical problem solving and code generation. We propose FIRE, a novel sampling method that integrates regularized sandbox execution verification directly into the autoregressive generation process. FIRE performs segment-wise intervention on response sequences and employs dynamic reweighting of token probabilities based on real-time execution feedback to prioritize high-confidence candidates. A key empirical finding is that the placement of execution interventions exhibits a non-monotonic impact on performance. Experiments demonstrate that FIRE improves answer correctness by 12.3% on mathematical reasoning benchmarks, accelerates alignment training convergence, and generalizes effectively across multiple open-source LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Since the release of ChatGPT, large language models (LLMs) have demonstrated remarkable capabilities across various domains. A key challenge in developing these general capabilities is efficiently sourcing diverse, high-quality data. This becomes especially critical in reasoning-related tasks with sandbox checkers, such as math or code, where the goal is to generate correct solutions to specific problems with higher probability. In this work, we introduce Flaming-hot Initiation with Regular Execution (FIRE) sampling, a simple yet highly effective method to efficiently find good responses. Our empirical findings show that FIRE sampling enhances inference-time generation quality and also benefits training in the alignment stage. Furthermore, we explore how FIRE sampling improves performance by promoting diversity and analyze the impact of employing FIRE at different positions within a response.

Problem

• Efficient sourcing of diverse, high-quality data
• Enhancing reasoning-related tasks in LLMs
• Improving inference-time generation quality

Innovation

• FIRE sampling enhances generation
• Promotes diversity in responses
• Improves training alignment stage