PhysX-CoT: Structured Physical Reasoning from a Single Image to Simulation-Ready 3D Assets

2026-08-08

AI Summary

This work addresses the limitations of existing methods for generating simulation-ready 3D assets from a single image, which suffer from implicit reasoning that entangles part layout with local shape and lacks supervisability over intermediate states. To overcome this, we propose an explicit, structured physical reasoning framework that models part decomposition, 2D/3D localization, inter-part relationships, coarse geometry, and surface cues sequentially through interpretable state trajectories, enabling supervision, conditional control, and optimization of intermediate steps. Our approach employs factorized decoding—using 3D bounding boxes for pose and local codes for shape—alongside a Chain-of-Thought-aligned GRPO algorithm and a frozen decoder architecture. Evaluated under a unified protocol, our method consistently outperforms baselines across geometric, scale, and physical plausibility metrics, producing assets that exhibit high-fidelity parseability, accurate collision responses, and functional articulation in Unreal Engine 5.

Abstract

Simulation-ready 3D assets are central to robotics and embodied AI. Generating them from a single image is usually framed as a vision-language model that emits a serialized asset for a decoder to turn into geometry and physical fields, leaving the image-to-3D reasoning implicit. We argue the limiting factor is this output-centric view: part placement and local shape are entangled in one global-coordinate token stream, and the intermediate physical states are never exposed for supervision, conditioning, or verification. PhysX-CoT instead casts single-image asset generation as an explicit structured physical reasoning process, an ordered and machine-parseable trajectory of part-level states covering decomposition, 2D and 3D grounding, relations, coarse geometry, and surface cues that we separately supervise, use to condition geometry, and treat as reward targets. Geometry is factorized so that 3D boxes carry placement and local codes carry shape, and CoT-aligned GRPO optimizes parse validity, grounding, geometry, placement, and physical consistency. Under a unified protocol that retrains all learned baselines on the same backbone, data, and frozen decoder, PhysX-CoT outperforms the closest full-task baseline across geometry, scale, and physical-attribute metrics. Oracle, token-matched, and state-order controls show the explicit states are functional rather than cosmetic, and in Unreal Engine~5 the generated assets parse, collide, and articulate at high validity.

Problem

• simulation-ready 3D assets
• single-image 3D generation
• physical reasoning
• structured representation
• intermediate state supervision

Innovation

• structured physical reasoning
• simulation-ready 3D assets
• chain-of-thought (CoT)
• factorized geometry representation
• GRPO optimization