ArgBench: Benchmarking LLMs on Computational Argumentation Tasks

2026-04-19

AI Summary

This work addresses the lack of a unified evaluation framework for large language models (LLMs) in computational argumentation by introducing ArgBench, the first comprehensive benchmark dedicated to this domain. ArgBench integrates 33 existing datasets spanning five core task categories—argument mining, stance assessment, argument quality judgment, argumentative reasoning, and argument generation—encompassing 46 distinct tasks in total. Through systematic experiments, the study evaluates five prominent LLM families under a consistent protocol and provides in-depth analysis of how factors such as in-context examples, chain-of-thought prompting, model scale, and training strategies influence argumentative capabilities. ArgBench establishes a standardized platform for evaluating computational argumentation systems and offers critical insights into the design choices that enhance LLMs’ performance on argument-related tasks.

Technology Category

• Knowledge Representation and Reasoning: Argumentation
• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Argumentation skills are an essential toolkit for large language models (LLMs). These skills are crucial in various use cases, including self-reflection, debating collaboratively for diverse answers, and countering hate speech. In this paper, we create the first benchmark for a standardized evaluation of LLM-based approaches to computational argumentation, encompassing 33 datasets from previous work in unified form. Using the benchmark, we evaluate the generalizability of five LLM families across 46 computational argumentation tasks that cover mining arguments, assessing perspectives, assessing argument quality, reasoning about arguments, and generating arguments. On the benchmark, we conduct an extensive systematic analysis of the contribution of few-shot examples, reasoning steps, model size, and training skills to the performance of LLMs on the computational argumentation tasks in the benchmark.

Problem

• computational argumentation
• benchmark
• large language models
• argumentation tasks
• standardized evaluation

Innovation

• computational argumentation
• benchmark
• large language models
• few-shot learning
• argument generation