Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement

2026-04-07

AI Summary

This work addresses the structural hallucinations in large language models during multi-token prediction, which arise from illicit shortcuts in the latent space and compromise the consistency of their internal world models. The authors theoretically analyze the gradient-induced biases inherent in multi-token prediction and propose Latent Semantic Enhancement for Multi-Token Prediction (LSE-MTP), a method that aligns discrete token predictions with continuous latent state representations by supervising the model with authentic latent state trajectories. By enforcing this alignment, LSE-MTP effectively mitigates structural hallucinations and substantially improves representation fidelity, consistency, and robustness to perturbations, as demonstrated on both synthetic graph data and real-world Manhattan taxi trajectory datasets.

Abstract

Whether Large Language Models (LLMs) develop coherent internal world models remains a core debate. While conventional Next-Token Prediction (NTP) focuses on one-step-ahead supervision, Multi-Token Prediction (MTP) has shown promise in learning more structured representations. In this work, we provide a theoretical perspective analyzing the gradient inductive bias of MTP, supported by empirical evidence, showing that MTP promotes the convergence toward internal belief states by inducing representational contractivity via gradient coupling. However, we reveal that standard MTP often suffers from structural hallucinations, where discrete token supervision encourages illegal shortcuts in latent space that violate environmental constraints. To address this, we propose a novel method Latent Semantic Enhancement MTP (LSE-MTP), which anchors predictions to ground-truth hidden state trajectories. Experiments on synthetic graphs and real-world Manhattan Taxi Ride show that LSE-MTP effectively bridges the gap between discrete tokens and continuous state representations, enhancing representation alignment, reducing structural hallucinations, and improving robustness to perturbations.

Problem

• world models
• multi-token prediction
• structural hallucinations
• latent representations
• environmental constraints

Innovation

• Multi-Token Prediction
• Latent Semantic Enhancement
• Structural Hallucination
• World Model Consistency
• Gradient Inductive Bias