LEXam: Benchmarking Legal Reasoning on 340 Law Exams

2025-05-19

AI Summary

Large language models (LLMs) exhibit weak structured, multi-step reasoning capabilities on long-form, open-ended legal examination questions. Method: We introduce LEXam—the first large-scale, multilingual, cross-curricular legal exam benchmark—comprising 4,886 English-German bilingual questions, accompanied by fine-grained, expert-verified reasoning path annotations. We propose an “LLM-as-a-Judge” evaluation paradigm integrating question-type classification, stepwise reasoning guidance, and legal expert validation into an automated scoring protocol, moving beyond answer accuracy alone. Contribution/Results: Experiments reveal systematic deficiencies in current state-of-the-art LLMs across core legal reasoning tasks—including element analysis and rule application—enabling fine-grained capability differentiation and reproducible, quantitative assessment of legal reasoning performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. We introduce LEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in particular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities. Adopting an LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how model-generated reasoning steps can be evaluated consistently and accurately. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. Project page: https://lexam-benchmark.github.io/

Problem

• Benchmarking long-form legal reasoning in LLMs using 340 law exams
• Evaluating LLMs on structured, multi-step legal reasoning challenges
• Assessing legal reasoning quality beyond simple accuracy metrics

Innovation

• LEXam benchmark with 340 law exams
• LLM-as-a-Judge for legal reasoning evaluation
• Multi-step legal reasoning guidance included