Open-KNEAD: Knowledge-grounded Nutrition Estimation via Agentic Decomposition

2026-07-14

AI Summary

Current retrieval-augmented dietary assessment methods struggle to simultaneously achieve portion accuracy, traceability, low user burden, interpretability, and privacy preservation within multimodal large language models. This work proposes a training-free, locally deployable agent framework that leverages nutrition-aware fine-grained retrieval to link each food item in meal images to authoritative nutritional databases (e.g., FNDDS), generating auditable, itemized nutrient records. By incorporating recipe priors to adjust for energy contributions from cooking ingredients, the approach mitigates systematic underestimation of non-American dishes. For the first time, it enables high-precision portion estimation and fully traceable documentation under minimal user burden and strict local execution. On the ACETADA dataset, the local agent surpasses two state-of-the-art closed-source models by approximately 30% and 53% in portion estimation accuracy, while ensuring all image data remains on-device throughout processing.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly used for dietary assessment from meal images, where retrieval-augmented grounding was shown to sharpen nutrition estimates. However, we find this premise no longer holds for current MLLMs. A modern MLLM's direct estimate now matches or surpasses the full retrieval pipeline. This raises a question: if retrieval no longer improves the overall estimate, can it still deliver the two things clinicians value, accurate portions and a traceable, item-by-item record? We pursue this while preserving what matters for clinical adoption: minimal user burden (a single, unannotated meal image), explainability (an auditable record), and privacy (locally hosted inference). We introduce Open-KNEAD, a knowledge-grounded agentic framework for meal nutrition estimation that is training-free and locally deployable. Each decomposed food item is grounded to a Food and Nutrient Database for Dietary Studies (FNDDS) code via selective, nutrient-aware retrieval, composing an auditable per-item record. Across two open MLLM families and three cuisines, Open-KNEAD improves portion estimates over both prior grounding methods and direct estimation in most backbone-dataset settings. An agent-internal recipe-prior step further recovers the invisible cooking-added energy that biases estimates on non-US cuisine. The advantage is largest on the dietitian-verified ACETADA dataset, where the local open agent surpasses the direct portion estimates of two frontier closed models by roughly $30\%$ and $53\%$, all while keeping every meal image on local hardware. We release the Open-KNEAD framework and its agent-ready FNDDS knowledge base.

Problem

• nutrition estimation
• dietary assessment
• portion accuracy
• explainability
• privacy-preserving

Innovation

• agentic decomposition
• knowledge-grounded retrieval
• nutrition estimation
• local deployment
• explainable AI