KitchenTwin: Semantically and Geometrically Grounded 3D Kitchen Digital Twins

2026-03-25

AI Summary

This work addresses the challenge of reliably fusing point clouds and local object meshes in monocular video-based 3D reconstruction, a problem exacerbated by scale ambiguity and inconsistent coordinate frames that hinder embodied AI training in semantically and geometrically consistent digital twins. To resolve this, we propose a scale-aware 3D fusion framework that leverages a vision-language model (VLM)-guided geometric anchoring mechanism to recover real-world metric scale. Our approach integrates gravity alignment, Manhattan world constraints, and collision-free local optimization to achieve physically plausible registration between global point clouds and object meshes. Key contributions include the first VLM-guided geometric anchor, a physics-constrained, geometry-aware registration pipeline, and the release of the first open-source indoor kitchen digital twin dataset featuring metric-scale accuracy and semantically registered object meshes, significantly improving cross-modal alignment and geometric consistency in real-world scenes.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Embodied AI training and evaluation require object-centric digital twin environments with accurate metric geometry and semantic grounding. Recent transformer-based feedforward reconstruction methods can efficiently predict global point clouds from sparse monocular videos, yet these geometries suffer from inherent scale ambiguity and inconsistent coordinate conventions. This mismatch prevents the reliable fusion of these dimensionless point cloud predictions with locally reconstructed object meshes. We propose a novel scale-aware 3D fusion framework that registers visually grounded object meshes with transformer-predicted global point clouds to construct metrically consistent digital twins. Our method introduces a Vision-Language Model (VLM)-guided geometric anchor mechanism that resolves this fundamental coordinate mismatch by recovering an accurate real-world metric scale. To fuse these networks, we propose a geometry-aware registration pipeline that explicitly enforces physical plausibility through gravity-aligned vertical estimation, Manhattan-world structural constraints, and collision-free local refinement. Experiments on real indoor kitchen environments demonstrate improved cross-network object alignment and geometric consistency for downstream tasks, including multi-primitive fitting and metric measurement. We additionally introduce an open-source indoor digital twin dataset with metrically scaled scenes and semantically grounded and registered object-centric mesh annotations.

Problem

• scale ambiguity
• coordinate inconsistency
• 3D digital twins
• metric geometry
• object-centric reconstruction

Innovation

• scale-aware 3D fusion
• Vision-Language Model (VLM)
• geometric anchoring
• metric digital twin
• object-centric reconstruction