🤖 AI Summary
This study addresses the challenges of legal information retrieval in Algeria, where high-quality corpora and relevance judgments are scarce, and manually constructing test collections is prohibitively expensive. Building upon the Cranfield paradigm, the authors propose a multi-stage automated retrieval and filtering approach combined with a semi-synthetic relevance judgment generation technique, which substantially reduces the need for human annotation. The method achieves a 99% reduction in annotation effort while preserving evaluation reliability, attaining a Hit@10 score of 0.785. System rankings demonstrate strong agreement with human assessments, as evidenced by Kendall’s τ of 0.89 and Spearman’s ρ of 0.92. This work offers a reproducible, low-cost framework for test collection construction in low-resource legal domains.
📝 Abstract
Test collections are essential for evaluating retrieval and re-ranking models. However, constructing such collections is challenging due to the high cost of manual annotation, particularly in specialized domains like Algerian legal texts, where high-quality corpora and relevance judgments are scarce. To address this limitation, we propose STCALIR, a framework for generating semi-synthetic test collections directly from raw legal documents. The pipeline follows the Cranfield paradigm, maintaining its core components of topics, corpus, and relevance judgments, while significantly reducing manual effort through automated multi-stage retrieval and filtering, achieving a 99% reduction in annotation workload. We validate STCALIR using the Mr. TyDi benchmark, demonstrating that the resulting semi-synthetic relevance judgments yield retrieval effectiveness comparable to human-annotated evaluations (Hit@10 \approx 0.785). Furthermore, system-level rankings derived from these labels exhibit strong concordance with human-based evaluations, as measured by Kendall's τ (0.89) and Spearman's \r{ho} (0.92). Overall, STCALIR offers a reproducible and cost-efficient solution for constructing reliable test collections in low-resource legal domains.