🤖 AI Summary
This study addresses the absence of a dedicated AI evaluation benchmark in French immigration law by introducing the first publicly available question-answering benchmark, encompassing three core administrative guidance tasks: residence permit type recommendation, required document retrieval, and citation of relevant legal provisions. Leveraging 52 annotated synthetic user profiles, the work systematically evaluates the performance of parametric large language models—including Qwen3.5-9B and Qwen3.5-27B—alongside dense retrieval-augmented generation (RAG) approaches across varying scales. Experimental results demonstrate that RAG substantially improves answer accuracy, particularly in permit recommendation, thereby underscoring the critical role of retrieval grounding in enhancing the reliability of legal AI systems.
📝 Abstract
International recruitment in France requires navigating a layered legal framework absent from existing legal AI benchmarks. We present a publicly available benchmark and first comparative evaluation for this domain, covering permit-type recommendation, required-document retrieval, and legal citation coverage. Comparing a parametric LLM baseline against dense retrieval augmentation at two model scales (Qwen3.5-9B and -27B) on 52 annotated synthetic profiles, we find that retrieval improves administrative guidance at both scales, most notably permit-type accuracy. Our results confirm that retrieval grounding is important for more reliable administrative guidance in this domain, and motivate further investigation of hybrid retrieval strategies.