Persuadability and LLMs as Legal Decision Tools

2026-04-28

AI Summary

This study investigates whether large language models (LLMs) in legal decision-making are susceptible to the quality of legal advocacy rather than relying solely on the substantive merits of a case. Through controlled experiments, the authors systematically evaluate how both state-of-the-art open-source and closed-source LLMs shift their positions and exhibit output consistency when presented with high- versus low-quality legal arguments. The work provides the first quantitative evidence of LLMs’ “persuadability” in legal contexts, revealing a significant tendency to adopt positions advanced by skillful advocates. These findings highlight a critical risk of advocacy-induced bias in judicial applications of current models and underscore the urgent need for improvements in model robustness and fairness.

Abstract

As Large Language Models (LLMs) are proposed as legal decision assistants, and even first-instance decision-makers, across a range of judicial and administrative contexts, it becomes essential to explore how they answer legal questions, and in particular the factors that lead them to decide difficult questions in one way or another. A specific feature of legal decisions is the need to respond to arguments advanced by contending parties. A legal decision-maker must be able to engage with, and respond to, including through being potentially persuaded by, arguments advanced by the parties. Conversely, they should not be unduly persuadable, influenced by a particularly compelling advocate to decide cases based on the skills of the advocates, rather than the merits of the case. We explore how frontier open- and closed-weights LLMs respond to legal arguments, reporting original experimental results examining how the quality of the advocate making those arguments affects the likelihood that a model will agree with a particular legal point of view, and exploring the factors driving these results. Our results have implications for the feasibility of adopting LLMs across legal and administrative settings.

Problem

• persuadability
• Large Language Models
• legal decision-making
• advocate quality
• judicial fairness

Innovation

• persuadability
• legal decision-making
• large language models
• argument quality
• AI in law