Is Energy Guidance All You Need? Training-Free Norm Injection for Driving World Models

2026-07-12

AI Summary

This work addresses the challenge of flexibly enforcing traffic rule constraints in video diffusion-based driving world models without retraining. The authors propose a training-free control method applied during the sampling phase, which injects driving regulations via differentiable energy functions to guide trajectory generation. Built upon the Open-Sora 2.0 MM-DiT rectified flow architecture, the approach jointly generates video and ego-vehicle trajectories and introduces a cross-stream coupling mechanism to enhance controllability. Experiments demonstrate that the method can effectively steer braking behavior toward counterfactual target locations, validating the feasibility of training-free control while also revealing limitations in current models regarding consistency between generated videos and vehicle trajectories.

Abstract

Driving world models built on large video-diffusion backbones generate realistic scenes but are hard to control: enforcing a traffic norm typically means retraining the backbone or conditioning it on hand-built layouts. We ask whether controllability requires training at all. Our experiment shows that a rectified-flow driving world model, which jointly generates future video and a planned ego trajectory, can have its planned trajectory steered entirely at sampling time by differentiable energy functions that encode driving norms, without knowledge-specific retraining of the diffusion backbone. Concretely, we demonstrate that a world model built on Open-Sora 2.0 MM-DiT backbone can be steered to brake at a counterfactual target by injecting energy guidance at sampling time. However, we find that the generated video does not yet follow the steered trajectory through the backbone's joint self-attention and identify the cross-stream coupling as a crucial requirement for end-to-end-controllable rollouts.

Problem

• controllability
• driving world models
• energy guidance
• video diffusion
• trajectory steering

Innovation

• energy guidance
• training-free control
• driving world models
• rectified flow
• cross-stream coupling