Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task

2026-04-26

AI Summary

This study addresses the lack of systematic evaluation of large language models (LLMs) on open-ended legal reasoning tasks, particularly within the Japanese judicial context. The authors construct the first dataset based on essay questions from the Japanese bar examination and introduce expert human evaluations to assess model-generated arguments, including both scoring and hallucination classification. Using this framework, they systematically evaluate mainstream LLMs’ abilities to identify legal issues, construct structured legal arguments, and characterize their hallucinatory tendencies. The findings reveal critical limitations of current models in handling such complex legal reasoning, while also providing a high-quality, expert-annotated benchmark that establishes a foundation for improving the reliability and interpretability of legal AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams. However, their capacity for generating open-ended legal reasoning in realistic scenarios remains insufficiently explored. Notably, to our best knowledge, there are no prior studies or datasets addressing this issue in the Japanese context. This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japanese jurisdiction. The dataset is based on the writing component of the Japanese bar examination, which requires examinees to identify multiple legal issues from long narratives and to construct structured legal arguments in free text format. Our key contribution is the manual evaluation of LLMs' generated responses by legal experts, which reveals limitations and challenges in legal reasoning. Moreover, we conducted a manual analysis of hallucinations to characterize when and how the models introduce content not supported by precedent or law. Our real exam questions, model-generated responses, and expert evaluations reveal the milestones of current LLMs in the Japanese legal domain. Our dataset and relevant resources will be available online.

Problem

• open-ended legal reasoning
• large language models
• Japanese bar exam
• legal evaluation
• hallucination

Innovation

• open-ended legal reasoning
• Japanese bar exam
• expert evaluation
• hallucination analysis
• legal LLM benchmark