Next-Token Prediction Task Assumes Optimal Data Ordering for LLM Training in Proof Generation

2024-10-30

arXiv.org

AI Summary

Large language models (LLMs) exhibit limited performance on theorem proving, partly because training data often orders proof steps by “logical verification order”—optimized for correctness checking—rather than “discovery order,” which aligns with human and model cognitive processes. Method: The authors formally define and empirically validate “intuitive order”: a stepwise ordering where each step is preceded by the minimal intermediate supervision signals required for its discovery. They train next-token prediction models under intuitive, logical, and random orderings, evaluating them on two controlled benchmarks—intuitionistic propositional logic proving and digit multiplication. Contribution/Results: Intuitive ordering improves proof success rate by 11 percentage points over logical ordering—the largest observed gap—significantly narrowing the performance gap. This work identifies proof-step ordering as a critical latent factor governing LLM theorem-proving capability, challenging the prevailing paradigm that emphasizes only data scale or model size.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In the field of large language model (LLM)-based proof generation, despite being trained on extensive corpora such as OpenWebMath and Arxiv, these models still exhibit only modest performance on proving tasks of moderate difficulty. We believe that this is partly due to the suboptimal order of each proof data used in training. Published proofs often follow a purely logical order, where each step logically proceeds from the previous steps based on the deductive rules. However, this order aims to facilitate the verification of the proof's soundness, rather than to help people and models learn the discovery process of the proof. In proof generation, we argue that the optimal order for one training data sample occurs when the relevant intermediate supervision for a particular proof step in the proof is always positioned to the left of that proof step. We call such order the intuitively sequential order. We validate our claims using two tasks: intuitionistic propositional logic theorem-proving and digit multiplication. Our experiments verify the order effect and provide support for our explanations. We demonstrate that training is most effective when the proof is in the intuitively sequential order. Moreover, the order effect and the performance gap between models trained on different data orders are substantial -- with an 11 percent improvement in proof success rate observed in the propositional logic theorem-proving task, between models trained on the optimal order compared to the worst order.

Problem

• Suboptimal data ordering hinders LLM proof generation performance
• Intuitively sequential order enhances proof step learning
• Optimal data ordering improves proof success rate significantly

Innovation

• Intuitively sequential order optimizes proof training data
• Optimal data ordering boosts proof success rate
• Identifies common order issues in math proofs