From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

2026-08-10

AI Summary

This study addresses the gap in existing large language model (LLM) evaluation frameworks, which often neglect public administration values and Dutch linguistic characteristics. Through expert consultation, user surveys, and civil servant interviews, the authors develop “Grip on LLMs,” a systematic evaluation framework tailored for Dutch government applications. It assesses over thirty general-purpose and Dutch-specific models across six dimensions: factuality, honesty, social bias, energy consumption, cost, and training data transparency. Innovatively integrating public governance principles with local language requirements, the work introduces a multi-dimensional trade-off perspective, revealing that factuality and honesty are governed by distinct mechanisms. A visualization tool is also developed to support non-technical decision-makers in model selection. Findings indicate no single model dominates across all criteria; high performance frequently entails greater environmental and economic costs, and bias shows no significant correlation with model capability. The results are publicly released as an accessible, multi-stakeholder model overview platform.

Abstract

Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts. We present the "Grip on LLMs" framework, a systematic evaluation suite for Dutch governmental use developed in collaboration with domain experts from a major Dutch municipal organisation. Through an advisory board process, user research, and a survey of the users of a civil-servant chatbot, we identify six evaluation dimensions (factuality, honesty, social bias, energy consumption, cost, and training data transparency) and operationalise them into a benchmark suite covering more than 30 multilingual and Dutch-specific models. Our results reveal that no single model excels across all dimensions, and that trade-offs are unavoidable: higher quality consistently comes at greater environmental impact and financial cost, while bias remains largely independent of both. We further find that factuality (whether a model answers correctly) and honesty (whether a model acknowledges what it does not know) are governed by distinct properties, with high factuality not implying high honesty. To make these findings actionable for non-technical audiences, we release a publicly accessible, user-friendly model overview designed for the full range of stakeholders involved in governmental LLM selection, from engineers to policymakers.

Problem

• large language models
• governmental use
• evaluation framework
• public administration values
• non-English contexts

Innovation

• evaluation framework
• governmental LLMs
• Dutch language
• factuality vs. honesty
• multilingual benchmark