PortionNet: Distilling 3D Geometric Knowledge for Food Nutrition Estimation

2025-12-25

AI Summary

Smartphones lack depth sensors, making accurate food volume and energy estimation from a single RGB image challenging. Method: We propose a cross-modal knowledge distillation framework that leverages point clouds to guide RGB network training for geometric representation learning; at inference, only RGB input is required. A dual-modality training strategy and a lightweight adapter network enable pseudo-3D perception. The architecture comprises a point cloud encoder, an RGB backbone, a feature alignment distillation module, and the adapter. Contribution/Results: Our method achieves state-of-the-art performance on MetaFood3D for both volume and energy estimation. Cross-dataset evaluation on SimpleFood45 demonstrates strong generalization in energy prediction. To our knowledge, this is the first work enabling high-precision, single-image nutritional inference without depth sensors—establishing a novel paradigm for mobile health dietary monitoring.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Accurate food nutrition estimation from single images is challenging due to the loss of 3D information. While depth-based methods provide reliable geometry, they remain inaccessible on most smartphones because of depth-sensor requirements. To overcome this challenge, we propose PortionNet, a novel cross-modal knowledge distillation framework that learns geometric features from point clouds during training while requiring only RGB images at inference. Our approach employs a dual-mode training strategy where a lightweight adapter network mimics point cloud representations, enabling pseudo-3D reasoning without any specialized hardware requirements. PortionNet achieves state-of-the-art performance on MetaFood3D, outperforming all previous methods in both volume and energy estimation. Cross-dataset evaluation on SimpleFood45 further demonstrates strong generalization in energy estimation.

Problem

• Estimates food nutrition from single RGB images
• Learns 3D geometry without depth sensors
• Generalizes across datasets for energy estimation

Innovation

• Distilling 3D geometric knowledge from point clouds
• Using cross-modal knowledge distillation for RGB-only inference
• Employing a dual-mode training strategy with adapter network