Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC

2026-06-14

AI Summary

This work addresses closed-loop motion planning and control using only visual inputs, with the goal of providing probabilistic safety guarantees. To this end, the authors propose a compact Markovian latent-state world model operating directly on pixel observations, integrated with System Level Synthesis (SLS)-based robust Model Predictive Control (MPC) for trajectory optimization. Notably, they introduce conformal prediction into the latent space for the first time, enabling a learnable and calibrated latent constraint checker that quantifies and bounds model errors. Evaluated on vision-based control tasks, the proposed method significantly outperforms existing baselines, demonstrating superior performance in both goal-reaching success rates and operational safety, thereby validating its effectiveness and reliability.

Abstract

We present SLS^2, a framework for safe feedback motion planning from pixels using robust model predictive control (MPC) in learned latent world models. Our approach trains an action-conditioned joint-embedding world model with compact Markovian latent states, enabling efficient gradient-based trajectory optimization through learned latent dynamics. To enforce safety for the true system despite imperfect latent predictions, we inform a GPU-accelerated system level synthesis (SLS) robust MPC scheme with conformal prediction to obtain calibrated latent error bounds and robust latent-space constraint sets. We further learn and conformalize a latent constraint checker, allowing the SLS planner to impose probabilistic safety constraints during closed-loop execution. We evaluate our method on vision-based control tasks, where it improves both goal-reaching performance and safety over latent world-model and safe-planning baselines.

Problem

• safe control
• latent world models
• vision-based planning
• probabilistic safety
• model predictive control

Innovation

• latent world model
• robust MPC
• conformal prediction
• system level synthesis
• safe reinforcement learning