SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics

2026-06-29

AI Summary

This work addresses the limitations of existing information retrieval (IR) benchmarks in accurately evaluating fine-grained relevance for mathematical content, which typically rely on costly manual annotations. To overcome this, the authors construct the first fully automated IR evaluation benchmark for mathematics, leveraging 283,000 high-school math problems. They employ large language models (LLMs) to extract solution summaries and topics, then combine ontology-based topic matching with lexical and semantic similarity to identify relevant documents. Fine-grained relevance scores are generated via a Swiss-system LLM preference tournament, eliminating the need for human annotation. Experiments reveal that modern embedding models substantially outperform both traditional and math-specific baselines, yet still struggle with symbol-intensive algebra and calculus tasks. Moreover, general-purpose benchmarks like MTEB fail to reliably predict retrieval performance in mathematical contexts, underscoring the necessity of domain-specific evaluation frameworks.

Abstract

As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to directly isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing relevant documents. We address this gap by introducing SABER-Math, the first fully automated benchmark for evaluating mathematical IR without expert annotation. Starting from 283K high-school-level math problems with solutions, SABER-Math builds challenging reranking tasks in three steps: (i) first, LLMs extract concise solution summaries and mathematical topics for each problem; (ii) then, per-query relevant documents are discovered using ontology topic-based and lexical solutions-summary-based similarities, and (iii) finally, a Swiss-style LLM preference tournament produces fine-grained relevance ratings for the documents. We evaluate lexical retrievers, specialized mathematical retrieval systems, and recent embedding models. We find that while modern embedding models substantially outperform classical and math-specific baselines, even the strongest systems struggle in symbol-heavy domains like Algebra and Calculus. Importantly, we show that general-purpose IR benchmarks such as MTEB do not reliably predict mathematical performance, especially for recent embedding models, highlighting the need for math-specific retrieval benchmarks.

Problem

• information retrieval
• mathematical relevance
• retrieval benchmark
• mathematical tasks
• evaluation

Innovation

• mathematical information retrieval
• automated benchmark
• LLM-based relevance annotation
• Swiss-style preference tournament
• embedding evaluation