HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulation

2026-08-13

AI Summary

This work addresses the need for clinical intelligence to infer latent patient states from incomplete observations, rather than merely learning isolated mappings from images to answers. To this end, the authors propose the first CT-centric multimodal world model that constructs a unified implicit representation of patient state, jointly handling three tasks: readout, reconstruction, and simulation. Built upon a 3-billion-parameter shared Transformer architecture, the model incorporates zero-initialized CT adapters and Hounsfield window sampling to enable bidirectional generation between language and volumetric data. Its effectiveness is validated on HounsBench, a newly introduced benchmark. Experimental results demonstrate superior performance across all three tasks, significantly advancing CT-based clinical understanding.

Abstract

Clinical intelligence requires estimating a patient's underlying condition from incomplete observations rather than learning isolated mappings from scans to answers. Volumetric medical images provide dense observations of anatomy, attenuation, and lesions, whereas clinical language provides sparse but complementary semantic observations. We formulate CT-centered intelligence as inference over a shared latent patient state, under which readout, reconstruction, and simulation all become state-dependent prediction problems. To operationalize this view, we introduce HounsBench, a computed tomography (CT) centric patient-state benchmark that unifies these three task families with patient-disjoint splits and per-family metrics, and HounsWorld, a 3B multimodal world model that treats volumetric scans and language as observations of the shared state through Joint Understanding-Generation Learning. A shared transformer forms an implicit patient-state estimate and supports three outputs: query-conditioned answers that read out the state, reports and captions that reconstruct it in language, and condition-specific CT volumes for low-dose denoising, virtual contrast enhancement, and anatomy-constrained text-and-mask-to-volume generation. Zero-initialized CT adapters preserve pretrained multimodal mappings, while condition-explicit Hounsfield-unit window sampling exposes clinically meaningful density observations. HounsWorld shows strong performance across all three task families while consistently improving CT understanding through clinically structured completion. Our project is available at https://github.com/byhwhite/HounsWorld.git

Problem

• patient-state estimation
• multimodal clinical intelligence
• CT-centered inference
• hidden state reconstruction
• medical image simulation

Innovation

• multimodal world model
• shared latent patient state
• joint understanding-generation learning
• Hounsfield-unit window sampling
• CT-centric clinical intelligence