A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences

2025-03-02

AI Summary

This paper addresses the limited interpretability and weak structured argumentation capabilities of large language models (LLMs) in legal reasoning. To this end, it proposes the first three-level hierarchical tree-based modeling paradigm tailored for legal reasoning: rooted at the factum probandum (ultimate factual claim), with downward branching into evidentiary chains and implicit judicial experience nodes. Methodologically, the work introduces a crowdsourced, structured legal reasoning dataset; develops a legal-knowledge-enhanced LLM; designs tree-structured generation modeling; implements a multi-stage reasoning agent; and incorporates an evidence–experience alignment mechanism. Key contributions include: (1) releasing the first benchmark for legal reasoning trees; (2) enabling auditable, bias-mitigated structured causal reasoning; and (3) significantly improving argument accuracy and reviewability—thereby establishing a novel AI-assisted adjudication paradigm for smart courts.

Technology Category

• Knowledge Representation and Reasoning: Argumentation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

While progress has been made in legal applications, law reasoning, crucial for fair adjudication, remains unexplored. We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience, enabling public scrutiny and preventing bias. Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision. We also create the first crowd-sourced dataset for this task, enabling comprehensive evaluation. Simultaneously, we propose an agent framework that employs a comprehensive suite of legal analysis tools to address the challenge task. This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the ``Intelligent Court''.

Problem

• Develops a law reasoning benchmark for LLMs with tree-structured elements.
• Introduces a task to convert case descriptions into hierarchical decision justifications.
• Creates a crowd-sourced dataset and proposes an agent framework for legal analysis.

Innovation

• Hierarchical law reasoning schema with factum probandum
• Crowd-sourced dataset for legal case evaluation
• Agent framework using legal analysis tools