A Reproducibility Study of Graph-Based Legal Case Retrieval

2025-04-11

AI Summary

This work addresses the poor reproducibility and limited generalizability of CaseLink, a graph neural network (GNN)-based legal case retrieval method. To this end, the authors conduct the first systematic reproduction and rigorous empirical validation of CaseLink, performing cross-domain transfer evaluation on both the original and a newly constructed benchmark dataset. They propose a multi-relational heterogeneous graph to model semantic associations between charges and cases, and integrate open-source large language models (LLMs) to enhance node representations and graph-structural understanding. Experimental results demonstrate substantial improvements in cross-dataset retrieval performance and robustness. All code, datasets, and experimental configurations are publicly released, establishing the first reproducible and extensible graph-based benchmark framework for legal information retrieval.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Legal retrieval is a widely studied area in Information Retrieval (IR) and a key task in this domain is retrieving relevant cases based on a given query case, often done by applying language models as encoders to model case similarity. Recently, Tang et al. proposed CaseLink, a novel graph-based method for legal case retrieval, which models both cases and legal charges as nodes in a network, with edges representing relationships such as references and shared semantics. This approach offers a new perspective on the task by capturing higher-order relationships of cases going beyond the stand-alone level of documents. However, while this shift in approaching legal case retrieval is a promising direction in an understudied area of graph-based legal IR, challenges in reproducing novel results have recently been highlighted, with multiple studies reporting difficulties in reproducing previous findings. Thus, in this work we reproduce CaseLink, a graph-based legal case retrieval method, to support future research in this area of IR. In particular, we aim to assess its reliability and generalizability by (i) first reproducing the original study setup and (ii) applying the approach to an additional dataset. We then build upon the original implementations by (iii) evaluating the approach's performance when using a more sophisticated graph data representation and (iv) using an open large language model (LLM) in the pipeline to address limitations that are known to result from using closed models accessed via an API. Our findings aim to improve the understanding of graph-based approaches in legal IR and contribute to improving reproducibility in the field. To achieve this, we share all our implementations and experimental artifacts with the community.

Problem

• Reproducing CaseLink's graph-based legal case retrieval method
• Assessing reliability and generalizability of CaseLink approach
• Improving reproducibility in graph-based legal IR research

Innovation

• Graph-based method models cases and charges as nodes
• Uses higher-order relationships beyond document level
• Incorporates open large language model for improved reliability