Did you just see that? Arbitrary view synthesis for egocentric replay of operating room workflows from ambient sensors
🤖 AI Summary
Traditional operating room (OR) observation relies on static surveillance cameras or subjective recall, failing to capture clinicians’ authentic egocentric perspectives—hindering surgical safety analysis, training, and workflow optimization. To address this, we propose a non-invasive egocentric viewpoint reconstruction framework that, for the first time, leverages existing fixed OR video streams and environmental sensor data to enable full-scene dynamic 3D reconstruction and arbitrary-viewpoint replay. Our method integrates multi-view geometric modeling with neural rendering and incorporates a diffusion model to enhance novel-view synthesis fidelity. Evaluated on multicenter surgical datasets, it achieves high-fidelity personalized field-of-view reconstruction and free-navigable viewpoint rendering—without requiring wearable devices or disrupting clinical workflows. This work transforms conventional OR surveillance systems into interactive 3D visual recording platforms, establishing a new paradigm for surgical cognition research, immersive training, and intelligent procedural analytics.
Application Category
📝 Abstract
Observing surgical practice has historically relied on fixed vantage points or recollections, leaving the egocentric visual perspectives that guide clinical decisions undocumented. Fixed-camera video can capture surgical workflows at the room-scale, but cannot reconstruct what each team member actually saw. Thus, these videos only provide limited insights into how decisions that affect surgical safety, training, and workflow optimization are made. Here we introduce EgoSurg, the first framework to reconstruct the dynamic, egocentric replays for any operating room (OR) staff directly from wall-mounted fixed-camera video, and thus, without intervention to clinical workflow. EgoSurg couples geometry-driven neural rendering with diffusion-based view enhancement, enabling high-visual fidelity synthesis of arbitrary and egocentric viewpoints at any moment. In evaluation across multi-site surgical cases and controlled studies, EgoSurg reconstructs person-specific visual fields and arbitrary viewpoints with high visual quality and fidelity. By transforming existing OR camera infrastructure into a navigable dynamic 3D record, EgoSurg establishes a new foundation for immersive surgical data science, enabling surgical practice to be visualized, experienced, and analyzed from every angle.
Problem
Research questions and friction points this paper is trying to address.
Reconstructs egocentric surgical viewpoints from fixed cameras
Enables arbitrary view synthesis without disrupting clinical workflow
Transforms operating room videos into navigable 3D records
Innovation
Methods, ideas, or system contributions that make the work stand out.
Geometry-driven neural rendering for viewpoint synthesis
Diffusion-based enhancement for visual fidelity
Reconstructing egocentric views from fixed cameras
🔎 Similar Papers
No similar papers found.
