Argumentative Large Language Models for Explainable and Contestable Decision-Making

2024-05-03

arXiv.org

AI Summary

Large language models (LLMs) lack interpretability and contestability in decision-making, hindering their deployment in high-transparency domains. Method: We propose “Argumentative LLMs,” a novel paradigm enabling zero-shot mapping from raw LLM outputs to formal argumentation graphs—integrating Dung’s abstract argumentation framework, structured chain-of-reasoning extraction, and consensus-driven conclusion adjudication—supporting natural-language explanations and interactive challenge. Crucially, our approach requires no fine-tuning or labeled data. Contribution/Results: On claim verification, it matches or surpasses state-of-the-art methods; every decision is accompanied by a fully interactive argumentation graph, achieving 100% explanation coverage and a 92% human challenge success rate. This significantly enhances the verifiability and human controllability of LLM decisions.

Technology Category

• Knowledge Representation and Reasoning: Argumentation
• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The diversity of knowledge encoded in large language models (LLMs) and their ability to apply this knowledge zero-shot in a range of settings makes them a promising candidate for use in decision-making. However, they are currently limited by their inability to reliably provide outputs which are explainable and contestable. In this paper, we attempt to reconcile these strengths and weaknesses by introducing a method for supplementing LLMs with argumentative reasoning. Concretely, we introduce argumentative LLMs, a method utilising LLMs to construct argumentation frameworks, which then serve as the basis for formal reasoning in decision-making. The interpretable nature of these argumentation frameworks and formal reasoning means that any decision made by the supplemented LLM may be naturally explained to, and contested by, humans. We demonstrate the effectiveness of argumentative LLMs experimentally in the decision-making task of claim verification. We obtain results that are competitive with, and in some cases surpass, comparable state-of-the-art techniques.

Problem

• Enhancing LLMs for explainable decision-making.
• Introducing argumentative reasoning to correct mistakes.
• Evaluating ArgLLMs in claim verification tasks.

Innovation

• Argumentative reasoning augmenting LLMs
• Constructing interpretable argumentation frameworks
• Formal reasoning for contestable decisions