VolE: A Point-cloud Framework for Food 3D Reconstruction and Volume Estimation

2025-05-15

AI Summary

Existing food volume estimation methods rely on specialized hardware (e.g., 3D scanners), depth sensors, or reference-object calibration—factors that hinder portability, calibration-free operation, and clinical-grade accuracy required in medical nutrition management. To address this, we propose the first reference-free, depth-free 3D food reconstruction and volumetric estimation framework tailored for off-the-shelf AR-enabled smartphones. Our method jointly leverages structure-from-motion (SfM) to reconstruct sparse point clouds, temporally consistent food instance segmentation from video, and geometrically constrained voxelization to enable end-to-end volume computation. To support robust segmentation under real-world complexity, we introduce the first large-scale food video segmentation dataset covering diverse, challenging scenarios. Evaluated across multiple benchmarks, our approach achieves a mean absolute percentage error (MAPE) of 2.22%, substantially outperforming prior art and enabling portable, calibration-free, clinically accurate nutritional assessment.

Technology Category

• Computer Vision: Segmentation
• Intelligent Robots: State Estimation
• Machine Learning: Feature Construction/Reformulation

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Accurate food volume estimation is crucial for medical nutrition management and health monitoring applications, but current food volume estimation methods are often limited by mononuclear data, leveraging single-purpose hardware such as 3D scanners, gathering sensor-oriented information such as depth information, or relying on camera calibration using a reference object. In this paper, we present VolE, a novel framework that leverages mobile device-driven 3D reconstruction to estimate food volume. VolE captures images and camera locations in free motion to generate precise 3D models, thanks to AR-capable mobile devices. To achieve real-world measurement, VolE is a reference- and depth-free framework that leverages food video segmentation for food mask generation. We also introduce a new food dataset encompassing the challenging scenarios absent in the previous benchmarks. Our experiments demonstrate that VolE outperforms the existing volume estimation techniques across multiple datasets by achieving 2.22 % MAPE, highlighting its superior performance in food volume estimation.

Problem

• Accurate food volume estimation for health monitoring
• Limitations of current methods using single-purpose hardware
• Mobile-based 3D reconstruction without reference objects

Innovation

• Mobile device-driven 3D reconstruction for food volume
• Reference- and depth-free framework using video segmentation
• AR-capable devices capture images for precise 3D models