Multimodal Generative Retrieval Model with Staged Pretraining for Food Delivery on Meituan

2026-02-06

AI Summary

This work addresses the “one-cycle problem” in multimodal retrieval, where joint optimization often leads to modality competition and training imbalance. To mitigate modality dominance and enable flexible training control, the authors propose a staged pretraining strategy that focuses on specific tasks at different phases. By integrating both generative and discriminative objectives, the approach enhances the model’s ability to capture semantic relationships among query, product features, and semantic IDs. Built upon a dual-tower architecture, the method fuses multimodal representations and incorporates semantic ID compression. Evaluated on large-scale Meituan data, it achieves consistent improvements of 3.80%/2.64%/2.17% in Recall@5/10/20 and 5.10%/4.22%/2.09% in NDCG@5/10/20. Online A/B tests further demonstrate a 1.12% increase in revenue and a 1.02% gain in click-through rate.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal retrieval models are becoming increasingly important in scenarios such as food delivery, where rich multimodal features can meet diverse user needs and enable precise retrieval. Mainstream approaches typically employ a dual-tower architecture between queries and items, and perform joint optimization of intra-tower and inter-tower tasks. However, we observe that joint optimization often leads to certain modalities dominating the training process, while other modalities are neglected. In addition, inconsistent training speeds across modalities can easily result in the one-epoch problem. To address these challenges, we propose a staged pretraining strategy, which guides the model to focus on specialized tasks at each stage, enabling it to effectively attend to and utilize multimodal features, and allowing flexible control over the training process at each stage to avoid the one-epoch problem. Furthermore, to better utilize the semantic IDs that compress high-dimensional multimodal embeddings, we design both generative and discriminative tasks to help the model understand the associations between SIDs, queries, and item features, thereby improving overall performance. Extensive experiments on large-scale real-world Meituan data demonstrate that our method achieves improvements of 3.80%, 2.64%, and 2.17% on R@5, R@10, and R@20, and 5.10%, 4.22%, and 2.09% on N@5, N@10, and N@20 compared to mainstream baselines. Online A/B testing on the Meituan platform shows that our approach achieves a 1.12% increase in revenue and a 1.02% increase in click-through rate, validating the effectiveness and superiority of our method in practical applications.

Problem

• multimodal retrieval
• modality imbalance
• one-epoch problem
• food delivery
• dual-tower architecture

Innovation

• staged pretraining
• multimodal retrieval
• generative retrieval
• semantic ID
• one-epoch problem