RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering

2026-04-22

AI Summary

This study addresses the scarcity of high-quality benchmarks for evaluating large language models (LLMs) on classical languages and cross-lingual reasoning tasks. We introduce the first Latin–English bilingual question-answering benchmark, comprising approximately 7,800 question-answer pairs derived from 19th-century textbooks, examinations, and quiz materials, encompassing knowledge-based, skill-based, multi-hop reasoning, and code-mixed questions. A reproducible data construction pipeline for low-resource languages is proposed, integrating automated extraction, cleaning, and human validation. We systematically evaluate state-of-the-art models—including LLaMA-3, Qwen QwQ, and OpenAI o3-mini—and find that current LLMs generally underperform on skill-intensive tasks such as metrical analysis and rhetorical identification. QwQ shows a slight advantage when prompted in Latin, while the performance of LLaMA-3 and o3-mini varies significantly across task types.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We introduce a benchmark dataset for question answering and translation in bilingual Latin and English settings, containing about 7,800 question-answer pairs. The questions are drawn from Latin pedagogical sources, including exams, quizbowl-style trivia, and textbooks ranging from the 1800s to the present. After automated extraction, cleaning, and manual review, the dataset covers a diverse range of question types: knowledge- and skill-based, multihop reasoning, constrained translation, and mixed language pairs. To our knowledge, this is the first QA benchmark centered on Latin. As a case study, we evaluate three large language models -- LLaMa 3, Qwen QwQ, and OpenAI's o3-mini -- finding that all perform worse on skill-oriented questions. Although the reasoning models perform better on scansion and literary-device tasks, they offer limited improvement overall. QwQ performs slightly better on questions asked in Latin, but LLaMa3 and o3-mini are more task dependent. This dataset provides a new resource for assessing model capabilities in a specialized linguistic and cultural domain, and the creation process can be easily adapted for other languages. The dataset is available at: https://github.com/slanglab/RespondeoQA

Problem

• Latin-English bilingual QA
• question answering benchmark
• low-resource language
• educational NLP
• classical language processing

Innovation

• bilingual QA benchmark
• Latin-English question answering
• low-resource language evaluation
• multihop reasoning
• educational NLP