MultiTEND: A Multilingual Benchmark for Natural Language to NoSQL Query Translation

2025-02-16

AI Summary

Existing NL2NoSQL approaches suffer from strong English dependency and inadequate multilingual support, causing a 4–6% performance drop in cross-lingual settings. To address this, we introduce MultiTEND—the first multilingual NL2NoSQL benchmark covering English, German, French, Russian, Japanese, and Chinese—and propose MultiLink, a novel framework featuring a pioneering parallel linking mechanism. This mechanism synergistically integrates multilingual collaborative encoding, chain-of-thought reasoning, and retrieval-augmented generation (RAG), augmented by cross-lingual semantic alignment and structured SQL decoding. On MultiTEND, MultiLink achieves ~15% higher execution accuracy on English and an average 10% improvement on non-English languages, consistently outperforming fine-tuned small models, zero-shot large language models, and RAG-based baselines. It significantly narrows the performance gap across languages, establishing a new state of the art in multilingual text-to-SQL translation.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Natural language interfaces for NoSQL databases are increasingly vital in the big data era, enabling users to interact with complex, unstructured data without deep technical expertise. However, most recent advancements focus on English, leaving a gap for multilingual support. This paper introduces MultiTEND, the first and largest multilingual benchmark for natural language to NoSQL query generation, covering six languages: English, German, French, Russian, Japanese and Mandarin Chinese. Using MultiTEND, we analyze challenges in translating natural language to NoSQL queries across diverse linguistic structures, including lexical and syntactic differences. Experiments show that performance accuracy in both English and non-English settings remains relatively low, with a 4%-6% gap across scenarios like fine-tuned SLM, zero-shot LLM, and RAG for LLM. To address the aforementioned challenges, we introduce MultiLink, a novel framework that bridges the multilingual input to NoSQL query generation gap through a Parallel Linking Process. It breaks down the task into multiple steps, integrating parallel multilingual processing, Chain-of-Thought (CoT) reasoning, and Retrieval-Augmented Generation (RAG) to tackle lexical and structural challenges inherent in multilingual NoSQL generation. MultiLink shows enhancements in all metrics for every language against the top baseline, boosting execution accuracy by about 15% for English and averaging a 10% improvement for non-English languages.

Problem

• Multilingual benchmark for NoSQL queries
• Challenges in multilingual query translation
• Framework for multilingual NoSQL generation

Innovation

• Multilingual Parallel Linking Process
• Chain-of-Thought reasoning integration
• Retrieval-Augmented Generation enhancement