DS@GT ARC at Touché: Large Language Models for Retrieval-Augmented Debate

2026-08-08

AI Summary

This study addresses two core tasks in retrieval-augmented debate: generating the next debating utterance and evaluating responses according to Gricean maxims—quantity, quality, relation, and manner. The authors propose a prompting pipeline that integrates retrieval-augmented generation (RAG) with an ensemble of multiple large language models (LLMs), systematically evaluating six state-of-the-art LLMs across three providers. Their empirical analysis, the first of its kind, reveals high intra-family consistency among models in applying Gricean criteria; however, this consensus shows limited correlation with official human ratings, particularly exhibiting significant deviation on the “quality” dimension. These findings illuminate both the promise and limitations of current LLMs for automated evaluation in argumentative contexts, offering new insights for the design of debate systems.

Abstract

We extend the DS@GT ARC working-note submission to the Touché 2025 Retrieval-Augmented Debate task. The task has two subtasks: generating the next utterance in a simulated debate, and evaluating debate responses according to the Gricean maxims of Quantity, Quality, Relation, and Manner. The DS@GT ARC submission consisted of six leading LLMs from three providers through a retrieval-augmented prompting pipeline. We summarize the results from the working paper and explore whether multi-LLM evaluator agreement is a reliable proxy for official evaluation performance. The analysis shows that frontier LLM systems are strong response generators, and as evaluators they agree strongly within model families. However this consensus does not reliably track the official evaluation target, with the largest gap on the Quality maxim. The accompanying source code for this paper is located at https://github.com/dsgt-arc/touche-2025-rad and https://github.com/dsgt-arc/touche-2025-rad-analysis.

Problem

• Retrieval-Augmented Debate
• Gricean maxims
• LLM evaluation
• debate response generation
• multi-LLM agreement

Innovation

• Retrieval-Augmented Debate
• Large Language Models
• Gricean Maxims
• Multi-LLM Evaluation
• Prompting Pipeline