J-LAW: Joint Localization and Actionable World Modeling via Coupled Latent Factor Graphs

2026-06-26

AI Summary

This work addresses the limitations of traditional SLAM, which lacks predictive planning capabilities, and action-conditioned world models, which often disregard metric consistency and suffer from open-loop drift. The paper presents the first unified formulation of SLAM and action-conditioned world modeling as a joint estimation problem, introducing a coupled factor graph framework. This framework enables bidirectional optimization between localization and actionable world representation through pose-latent coupling factors and supports latent loop closure. By integrating multi-source probabilistic constraints—including observations, action predictions, odometry, latent landmarks, and loop closures—the approach significantly reduces latent prediction RMSE and end-point drift on real-world PushT and WildGS datasets, while generating globally consistent maps that preserve both metric accuracy and actionability.

Abstract

Classical SLAM estimates metric poses and a geometric map but produces no actionable predictive model for planning. Action-conditioned world models learn compact latent dynamics for planning but ignore global metric consistency and accumulate drift under open-loop rollout. We argue these are two views of the same estimation problem and propose J-LAW (Joint Localization and Actionable World Modeling) in this letter: a coupled factor graph that jointly optimizes metric object poses, latent world states, and latent landmark embeddings. The bridge is a pose-conditioned latent encoder and a learned pose--latent coupling factor, so that better localization improves the world model and vice versa. We cast observation, action-conditioned prediction, metric odometry, pose--latent coupling, latent loop closure, and latent landmark observation as probabilistic factors in a single MAP objective. Real-data experiments on PushT and WildGS show that coupled graph correction substantially reduces latent prediction RMSE and endpoint drift relative to open-loop rollout, while latent loop closure improves global trajectory consistency. J-LAW yields a map that is simultaneously metric (poses) and actionable (latent landmarks for planning).

Problem

• SLAM
• actionable world modeling
• metric consistency
• latent dynamics
• pose estimation

Innovation

• J-LAW
• factor graph
• actionable world modeling
• latent dynamics
• metric SLAM