Internalizing the Future: A Unified Agentic Training Paradigm for World Model Planning

2026-06-25

AI Summary

This work addresses the challenge that large language model agents struggle with effective forward-looking planning in long-horizon tasks due to the absence of an internalized world model. To overcome this limitation, the authors propose a capability-first, three-stage training framework comprising World-Model Agent Mid-training (WM-AMT), Format-Guided Supervised Fine-Tuning (FE-SFT), and Foresight-Conditioned Reinforcement Learning (FC-RL). This framework enables a single autoregressive model to generate coherent future state rollouts and calibrate the likelihood of plan success. Evaluated on search and mathematical reasoning tasks, the approach significantly outperforms existing baselines, achieving— for the first time—a well-calibrated internal world model with genuine predictive capacity, thereby effectively bridging the gap between output format alignment and planning competence.

Abstract

Large language model (LLM) agents have demonstrated strong capability in sequential decision-making, yet they remains fundamentally reactive in long-horizon tasks. Unlike humans who employ "what-if" reasoning to evaluate potential plans before commitment, standard agents lack an internal world model to simulate future outcomes. Therefore, we propose to internalize future-aware planning by training a single autoregressive model to verbalize both a prospective state rollout and a plan-conditioned success estimate-a textual analogue of the Q-value. Crucially, we identify a format-capability gap: simply fine-tuning agents on look-ahead traces during post-training leads to superficial mimicry of foresight without genuine predictive grounding. To bridge this gap, we introduce a three-stage training paradigm: (i) World Model Agentic Mid-Training (WM-AMT) to inject latent predictive capabilities into the policy; (ii) Format-Eliciting SFT (FE-SFT) to structure this injected capability; and (iii) Foresight-Conditioned Reinforcement Learning (FC-RL) to refine the calibration and utility of the generated simulations. Evaluated on search and mathematical reasoning tasks, our approach consistently outperforms other training baselines. Our results demonstrate that effective internal world modeling in LLM agents requires a capability-first training pipeline to achieve grounded and calibrated foresight.

Problem

• world model
• foresight
• long-horizon planning
• internal simulation
• prospective reasoning

Innovation

• world model
• foresight planning
• agentic training
• format-capability gap
• autoregressive simulation