MARCA: A Checklist-Based Benchmark for Multilingual Web Search

2026-04-15

AI Summary

This work addresses the lack of systematic evaluation benchmarks for large language models in multilingual (particularly Portuguese) web search, evidence selection, and answer synthesis. To bridge this gap, we introduce MARCA—the first bilingual benchmark for English and Portuguese—comprising 52 human-crafted multi-entity questions accompanied by checklist-based scoring rubrics. MARCA incorporates run-level uncertainty metrics and a subtask decomposition mechanism, evaluated through direct web search and an Orchestrator agent framework. Experiments across 14 models demonstrate that task orchestration substantially improves answer coverage and reveal significant performance disparities between English and Portuguese, thereby validating MARCA’s effectiveness and necessity for multilingual model assessment.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesize complete answers. While recent benchmarks evaluate web-browsing and agentic tool use, multilingual settings, and Portuguese in particular, remain underexplored. We present \textsc{MARCA}, a bilingual (English and Portuguese) benchmark for evaluating LLMs on web-based information seeking. \textsc{MARCA} consists of 52 manually authored multi-entity questions, paired with manually validated checklist-style rubrics that explicitly measure answer completeness and correctness. We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents. To capture stochasticity, each question is executed multiple times and performance is reported with run-level uncertainty. Across models, we observe large performance differences, find that orchestration often improves coverage, and identify substantial variability in how models transfer from English to Portuguese. The benchmark is available at https://github.com/maritaca-ai/MARCA

Problem

• multilingual web search
• large language models
• benchmark
• Portuguese
• information seeking

Innovation

• multilingual web search
• checklist-based evaluation
• LLM benchmarking
• task orchestration
• answer completeness