FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation

2024-09-27

Conference on Multimedia Modeling

AI Summary

This work addresses the task of Japanese recipe generation—a culturally grounded, low-resource multimodal generation problem. We propose the first end-to-end multimodal large language model (MLLM) framework tailored to Japanese culinary culture. Methodologically, we extend the LLaVA architecture by integrating the Japanese-optimized LLaMA language model with a ResNet-50 visual encoder, and introduce two domain-specific innovations: (i) cuisine-aware visual prompt tuning and (ii) dish-name–step alignment constraints. Additionally, we employ culinary knowledge–enhanced instruction fine-tuning. Evaluated on our newly constructed JP-RecipeBench benchmark, our approach achieves a +12.6 BLEU-4 improvement over baselines; human evaluation confirms 87% of generated recipes are both operationally feasible and culturally appropriate. To our knowledge, this is the first system capable of generating structured, idiomatic Japanese cooking instructions directly from food images. The work establishes a reusable technical paradigm and publicly releases curated data resources for multilingual, domain-specific multimodal generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Problem

• Improving Japanese recipe generation using Multimodal Large Language Models.
• Enhancing food image understanding through recipe data in Japanese.
• Benchmarking open MLLMs against GPT-4o for recipe accuracy.

Innovation

• Fine-tuned LLaVA-1.5 and Phi-3 Vision models
• Benchmarked against GPT-4o for recipe generation
• Achieved higher F1 score in ingredient generation