SemBench: A Universal Semantic Framework for LLM Evaluation

2026-03-12

AI Summary

Current evaluations of semantic understanding in large language models (LLMs) rely on manually curated benchmarks, which are costly to develop and difficult to scale across languages. This work proposes SemBench, a framework that, for the first time, automatically generates language-agnostic semantic similarity evaluation data using only dictionary definitions and sentence encoders—eliminating the need for human-annotated example sentences. The approach substantially reduces benchmark construction costs and enables support for low-resource languages. Evaluated on English, Spanish, and Basque, SemBench demonstrates high correlation with the standard Word-in-Context (WiC) benchmark and achieves stable model rankings with only a small number of samples.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent progress in Natural Language Processing (NLP) has been driven by the emergence of Large Language Models (LLMs), which exhibit remarkable generative and reasoning capabilities. However, despite their success, evaluating the true semantic understanding of these models remains a persistent challenge. Traditional benchmarks such as Word-in-Context (WiC) effectively probe this capability, but their creation is resource-intensive and often limited to high-resource languages. In this paper, we introduce SemBench, a framework for automatically generating synthetic benchmarks that assess the semantic competence of LLMs using only dictionary sense definitions and a sentence encoder. This approach eliminates the need for curated example sentences, making it both scalable and language-independent. We evaluate SemBench in three languages (English, Spanish, and Basque) spanning different levels of linguistic resources, and across a wide range of LLMs. Our results show that rankings derived from SemBench strongly correlate with those obtained from standard WiC datasets. Furthermore, our analysis demonstrates that only a small number of examples is required to achieve stable and meaningful rankings. Overall, SemBench provides a lightweight, adaptable, and data-efficient framework for cross-lingual evaluation of semantic understanding in LLMs.

Problem

• semantic understanding
• LLM evaluation
• cross-lingual benchmark
• low-resource languages
• semantic competence

Innovation

• semantic evaluation
• synthetic benchmark
• language-independent
• dictionary-based generation
• cross-lingual LLM assessment