PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese

2025-11-21

AI Summary

To address the lack of dedicated evaluation frameworks for Portuguese large language models (LLMs) and the limited understanding of cross-lingual performance disparities, this work introduces PoETa v2—the first large-scale, multi-task Portuguese benchmark, comprising 40+ natural language understanding, generation, and reasoning tasks. Leveraging PoETa v2, we systematically evaluate over 20 open- and closed-source LLMs across diverse parameter scales and computational budgets, and conduct the first rigorous Portuguese–English cross-lingual performance analysis. Results reveal persistent deficiencies in linguistic adaptation and cultural contextual modeling; moreover, computational investment and language-specific fine-tuning exhibit a nonlinear interaction effect on performance gains. PoETa v2 is publicly released to support fair, reproducible, and linguistically grounded evaluation of non-English LLMs.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse languages. In this work, we present the most extensive evaluation of LLMs for the Portuguese language to date. Leveraging our newly introduced PoETa v2 benchmark -- a comprehensive suite of over 40 tasks in Portuguese -- we assess more than 20 models covering a broad spectrum of training scales and computational resources. Our study reveals how computational investment and language-specific adaptation impact performance in Portuguese, while also analyzing performance gaps in comparison to equivalent tasks in English. Through this benchmark and analysis, PoETa v2 lays the groundwork for future research on Portuguese language modeling and evaluation. The benchmark is available at https://github.com/PoETaV2/PoETaV2.

Problem

• Evaluating LLM performance variations in Portuguese linguistic contexts
• Assessing over 20 models using comprehensive Portuguese benchmark suite
• Analyzing computational investment and language adaptation impacts on Portuguese

Innovation

• Introduced PoETa v2 benchmark for Portuguese evaluation
• Assessed over 20 models across 40+ Portuguese tasks
• Analyzed computational investment and language adaptation impacts