Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

2026-06-22

AI Summary

This study addresses the issue of large language models (LLMs) frequently triggering safety guardrails in multilingual criminal legal contexts due to over-alignment, leading to unwarranted refusals of legitimate tasks or insertion of disclaimers that undermine the reliability of functions such as translation and summarization. The work presents the first systematic quantification of this problem, revealing its complex dependence on model-language interactions, and introduces TF-RefusalBench—a multilingual benchmark comprising 5,200 prompts derived from rulings of the Swiss Federal Supreme Court. By integrating prompt engineering with abliteration (a technique that ablates refusal-inducing directions in model representations), the authors propose an intervention strategy that balances task fidelity and safety. Experiments demonstrate that abliteration effectively suppresses refusal behaviors with negligible impact on task performance, significantly outperforming approaches relying solely on prompt adjustments.

Abstract

While the wider applicability of LLMs in the legal field is currently debated due to their reliability and the gravity of any errors, narrow uses with well-understood and mitigated risks have emerged. Notably the Swiss Federal Supreme Court uses small on-premises models for tentative translations and short-passage summarization across the four official languages. However, such usage is challenging in the context of Criminal Law. Since rulings and cases employees work on routinely can contain detailed descriptions of violent and sexual offenses, their legitimate work is compromised by refusals and disclaimers due to the activation of model guardrails (over-alignment). To measure this phenomenon, we introduce TF-RefusalBench, a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. TF-RefusalBench contains 5,200 total prompts across French, German, Italian, and English, corresponding to common task prompts and passages likely to trigger refusal. We then use TF-RefusalBench to show that over-alignment is a multifaceted phenomenon, influenced by the model and the prompt and text languages being processed, and that its impact cannot be evaluated solely from an over-refusal perspective, given the disclaimer's impact on task faithfulness. Finally, we evaluate approaches to enable on-premises LLMs for Criminal Law Tasks, demonstrating that while prompting can be effective, abliteration (refusal directions ablation) eliminates refusal with minimal impact on task performance.

Problem

• over-alignment
• LLMs
• criminal law
• multilingual
• refusal

Innovation

• over-alignment
• TF-RefusalBench
• multilingual legal NLP
• refusal ablation
• on-premises LLMs