DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

2026-06-13

AI Summary

This work addresses the limitation of existing multimodal reasoning methods, which typically assume a single interpretation for each input and employ fixed reasoning paths or uniform computational budgets, thereby struggling with ambiguities arising from occlusion, blur, viewpoint variation, or semantic uncertainty. To overcome this, the authors propose a diversity-aware reasoning framework grounded in a continuous latent space. The approach maintains multiple distinct latent-world hypotheses through orthogonality regularization, performs independent reasoning over each hypothesis, and formulates the overall process as a resource-constrained sequential decision problem. A reinforcement learning agent dynamically allocates computational resources to adaptively expand, terminate, or merge reasoning paths. Evaluated on multiple multimodal benchmarks, the method outperforms state-of-the-art approaches by 2–5 percentage points in accuracy while reducing memory consumption by 24%.

Abstract

Reasoning capabilities of multimodal large language models (MLLMs) have improved considerably in recent years. Existing approaches typically rely on explicit chain-of-thought or continuous latent-space trajectories to enhance multi-step reasoning. However, these methods generally assume that an input admits a single latent interpretation and unfold reasoning along a fixed path or under a uniform computation budget. In real-world multimodal settings, visual observations are often subject to occlusion, blur, viewpoint variation, or semantic ambiguity, giving rise to multiple plausible interpretations. A uniform reasoning strategy not only limits the model's ability to explore multiple hypotheses but also incurs high memory usage and rollout cost. We present DLWM (Diverse Latent World Models), a multimodal reasoning framework that combines latent-space reasoning with reinforcement learning. First, we construct a set of diverse latent world hypotheses in continuous latent space, each capturing a different plausible interpretation of the visual input, and unfold latent reasoning independently on each hypothesis. An orthogonality-based diversity regularizer explicitly prevents hypothesis collapse. Second, we formulate the latent reasoning process as a resource-constrained sequential decision problem and introduce a resource-aware reinforcement learning policy that adaptively allocates computation across hypotheses, dynamically deciding whether to expand, terminate, or merge reasoning paths, thereby substantially reducing memory footprint and improving rollout efficiency. Experiments on multiple multimodal reasoning benchmarks demonstrate that DLWM outperforms existing methods by 2-5 points in accuracy while reducing memory usage by 24%.

Problem

• multimodal reasoning
• latent interpretation
• hypothesis diversity
• computation budget
• visual ambiguity

Innovation

• Diverse Latent World Models
• Multimodal Reasoning
• Latent Space Diversity
• Resource-Aware Reinforcement Learning
• Hypothesis Exploration