🤖 AI Summary
Existing multimodal retrieval platforms struggle to simultaneously achieve transparent data storage, rich hybrid querying, expressive semantic modeling, and high query efficiency. To address this, we propose a data-lake-oriented multimodal retrieval platform featuring a novel query-aware multimodal feature representation mechanism and an end-to-end differentiable learned indexing architecture. This enables cross-modal semantic alignment and joint vector-structured querying. The platform integrates open APIs, query-driven encoding networks, and high-dimensional learned indexes, supporting millisecond-scale cross-modal semantic retrieval across text, images, and tabular data. Experimental evaluation demonstrates that, on rich hybrid query workloads, our platform achieves a 2.3× throughput improvement and a 57% latency reduction compared to conventional vector databases and multimodal database systems—significantly advancing state-of-the-art performance in multimodal retrieval.
📝 Abstract
Multimodal data has become a crucial element in the realm of big data analytics, driving advancements in data exploration, data mining, and empowering artificial intelligence applications. To support high-quality retrieval for these cutting-edge applications, a robust multimodal data retrieval platform should meet the challenges of transparent data storage, rich hybrid queries, effective feature representation, and high query efficiency. However, among the existing platforms, traditional schema-on-write systems, multi-model databases, vector databases, and data lakes, which are the primary options for multimodal data retrieval, make it difficult to fulfill these challenges simultaneously. Therefore, there is an urgent need to develop a more versatile multimodal data retrieval platform to address these issues. In this paper, we introduce a Multimodal Data Retrieval Platform with Query-aware Feature Representation and Learned Index based on Data Lake (MQRLD). It leverages the transparent storage capabilities of data lakes, integrates the multimodal open API to provide a unified interface that supports rich hybrid queries, introduces a query-aware multimodal data feature representation strategy to obtain effective features, and offers high-dimensional learned indexes to optimize data query. We conduct a comparative analysis of the query performance of MQRLD against other methods for rich hybrid queries. Our results underscore the superior efficiency of MQRLD in handling multimodal data retrieval tasks, demonstrating its potential to significantly improve retrieval performance in complex environments. We also clarify some potential concerns in the discussion.