BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams

2026-06-21

AI Summary

This study addresses the lack of systematic evaluation benchmarks for large language models (LLMs) on open-ended Portuguese-language tasks, such as free-response questions from university entrance exams. To bridge this gap, the authors introduce BLUEX v2, a novel benchmark comprising 395 exam questions and 919 sub-questions from Brazil’s top universities, UNICAMP and USP, enriched with images, reference answers, grading rubrics, and cognitive skill labels—the first dataset of its kind to incorporate such multimodal and multidimensional annotations. Using an LLM-as-a-judge methodology, the authors evaluate 21 prominent LLMs, yielding scores between 4.18 and 9.10 (out of 10) and revealing persistent weaknesses in mathematical reasoning and image understanding. This work establishes the first comprehensive framework for evaluating high-level reasoning in Portuguese, offering a foundation for future research in multilingual and multimodal assessment.

Abstract

Although Large Language Models (LLMs) excel in many tasks, their assessment in Portuguese has received less attention, particularly for open-ended, discursive tasks that demand deeper reasoning and generation capabilities. While the original BLUEX benchmark addressed the scarcity of Portuguese evaluation datasets through multiple-choice questions from Brazilian university entrance exams, it did not cover the more challenging second-phase examinations, which require free-form written responses. In this work, we introduce BLUEX v2, a benchmark derived from the second-phase entrance exams of Brazil's two leading universities: UNICAMP (Comvest) and USP (Fuvest), spanning exam years 2022-2025. Our dataset comprises 395 questions unfolding into 919 graded subquestions, with 55.7% of questions containing associated images. Each question is annotated with subject area, official reference answers, LLM-generated rubric criteria, and six cognitive capability tags. We evaluate 21 state-of-the-art LLMs using an LLM-as-a-judge protocol. Results reveal a 4.92-point performance spread across models (4.18-9.10 on a 0-10 scale), with Mathematical Reasoning and Image Understanding emerging as the hardest capability dimensions. The dataset, evaluation code, and model outputs are publicly available at https://anonymous.4open.science/r/BLUEXv2.

Problem

• Large Language Models
• Portuguese language
• open-ended questions
• Brazilian university entrance exams
• discursive tasks

Innovation

• open-ended benchmark
• Portuguese LLM evaluation
• LLM-as-a-judge
• multimodal reasoning
• cognitive capability tagging