LEDOM: An Open and Fundamental Reverse Language Model

2025-07-01

AI Summary

This work addresses the inherent limitation of unidirectional temporal modeling in standard language models by proposing LEDOM—the first purely reverse-language model—trained to autoregressively predict preceding tokens via inverse-time sequence processing. Methodologically, we construct 2B- and 7B-parameter reverse foundation models trained on 435 billion tokens and introduce Reverse Reward, a novel mechanism that leverages LEDOM to perform posterior re-ranking of forward-model outputs. Our key contributions are: (1) demonstrating that reverse-language models serve as effective general-purpose foundation models, exhibiting distinctive posterior reasoning capabilities—especially in mathematical reasoning; (2) significantly improving mathematical reasoning performance of existing forward models *without any architectural modification*, solely through LEDOM-guided re-ranking; and (3) open-sourcing all models, code, and data. Empirical results validate reverse-time modeling as a viable and promising new paradigm for language model architecture design and reasoning methodology.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We introduce LEDOM, the first purely reverse language model, trained autoregressively on 435B tokens with 2B and 7B parameter variants, which processes sequences in reverse temporal order through previous token prediction. For the first time, we present the reverse language model as a potential foundational model across general tasks, accompanied by a set of intriguing examples and insights. Based on LEDOM, we further introduce a novel application: Reverse Reward, where LEDOM-guided reranking of forward language model outputs leads to substantial performance improvements on mathematical reasoning tasks. This approach leverages LEDOM's unique backward reasoning capability to refine generation quality through posterior evaluation. Our findings suggest that LEDOM exhibits unique characteristics with broad application potential. We will release all models, training code, and pre-training data to facilitate future research.

Problem

• Introducing LEDOM as the first reverse language model
• Exploring reverse language model's foundational potential for general tasks
• Using LEDOM for Reverse Reward to improve mathematical reasoning

Innovation

• Reverse temporal order autoregressive training
• Reverse Reward for model output reranking
• Backward reasoning capability for quality refinement