Toward Human Rights Benchmarking for LLMs: A Pilot Methodology

2026-08-10

AI Summary

This study addresses the absence of effective benchmarks for evaluating large language models’ (LLMs’) reasoning capabilities in human rights law. It introduces HumRightsBench, the first evaluation benchmark grounded in real-world scenarios and validated by human rights experts. The work innovatively integrates the structural obligations of international human rights law into LLM assessment, employing a refined IRAP legal reasoning framework—replacing the traditional IRAC—to formulate evaluation heuristics. Furthermore, it establishes a scalable methodology by incorporating diverse, globally sourced human rights practitioner-annotated contexts. Experimental results demonstrate substantial performance disparities among mainstream LLMs across tasks, with overall accuracy ranging from 0.339 to 0.577 and task-level accuracy spanning 0.025 to 0.774, thereby confirming both the validity and necessity of the proposed benchmark.

Abstract

Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights law. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench: the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of international human rights law. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that model accuracy scores range considerably across legal reasoning tasks (overall model performance ranges from 0.339 to 0.577, task min-max ranges from 0.025 to 0.774), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical moment in its evolution.

Problem

• human rights
• large language models
• legal reasoning
• evaluation benchmark
• AI alignment

Innovation

• HumRightsBench
• IRAP framework
• human rights reasoning
• scenario-based benchmark
• LLM evaluation