BRoverbs -- Measuring how much LLMs understand Portuguese proverbs

2025-09-10

AI Summary

Existing Portuguese evaluation datasets predominantly rely on translation or focus narrowly on structured exams and social media, failing to capture linguistic nuance and cultural context—particularly the comprehension of non-literal expressions such as proverbs. To address this gap, we introduce BRoverbs, the first native evaluation benchmark for Brazilian Portuguese proverbs. BRoverbs is designed to assess cultural semantic parsing, metaphor understanding, and multi-task language reasoning, thereby overcoming limitations of prior resources. Constructed exclusively from authentic, culturally grounded Brazilian proverbs, the benchmark emphasizes cultural fidelity and deep language understanding, enabling scalable and quantifiable regional model evaluation. BRoverbs is publicly released to advance robustness and cultural adaptability of large language models in Portuguese-speaking contexts. It serves as a foundational tool for building culturally intelligent evaluation frameworks for low-resource languages.

Technology Category

• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Provenance, trust, security and privacy, and ethical issues in managing semantic data

Abstract

Large Language Models (LLMs) exhibit significant performance variations depending on the linguistic and cultural context in which they are applied. This disparity signals the necessity of mature evaluation frameworks that can assess their capabilities in specific regional settings. In the case of Portuguese, existing evaluations remain limited, often relying on translated datasets that may not fully capture linguistic nuances or cultural references. Meanwhile, native Portuguese-language datasets predominantly focus on structured national exams or sentiment analysis of social media interactions, leaving gaps in evaluating broader linguistic understanding. To address this limitation, we introduce BRoverbs, a dataset specifically designed to assess LLM performance through Brazilian proverbs. Proverbs serve as a rich linguistic resource, encapsulating cultural wisdom, figurative expressions, and complex syntactic structures that challenge the model comprehension of regional expressions. BRoverbs aims to provide a new evaluation tool for Portuguese-language LLMs, contributing to advancing regionally informed benchmarking. The benchmark is available at https://huggingface.co/datasets/Tropic-AI/BRoverbs.

Problem

• Evaluating LLM understanding of Portuguese proverbs
• Assessing cultural and linguistic nuances in Portuguese LLMs
• Addressing limited native Portuguese evaluation datasets

Innovation

• Native Brazilian proverb dataset creation
• Evaluating cultural and linguistic nuance comprehension
• Region-specific Portuguese LLM benchmarking tool