Judgment-of-Thought Prompting: A Courtroom-Inspired Framework for Binary Logical Reasoning with Large Language Models

2024-09-25

AI Summary

Large language models (LLMs) exhibit limited performance on binary logical reasoning tasks, and existing prompting methods struggle to ensure both reasoning accuracy and interpretability. To address this, we propose Judgment-of-Thought (JoT), a courtroom-inspired multi-agent prompting framework that orchestrates three specialized roles—Lawyer (generating arguments), Prosecutor (raising counterarguments), and Judge (rendering final verdicts)—to enable structured, iterative debate. JoT innovatively integrates role-based prompting, cross-model feedback fusion, and argument refinement into prompt engineering. This design enhances reasoning robustness and transparency. Evaluated on BigBench-Hard and Winogrande, JoT achieves state-of-the-art performance, attaining 98% accuracy on Boolean expression reasoning. Ablation studies confirm the critical contributions of role specialization and iterative feedback mechanisms to overall efficacy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper proposes a novel prompting approach, Judgment of Thought (JoT), specifically tailored for binary logical reasoning tasks. Despite advances in prompt engineering, existing approaches still face limitations in handling complex logical reasoning tasks. To address these issues, JoT introduces a multi-agent approach with three specialized roles$unicode{x2010}$$unicode{x2010}$$unicode{x2010}$lawyer, prosecutor, and judge$unicode{x2010}$$unicode{x2010}$$unicode{x2010}$where a high-level model acts as the judge, and lower-level models serve as lawyer and prosecutor to systematically debate and evaluate arguments. Experimental evaluations on benchmarks such as BigBenchHard and Winogrande demonstrate JoT's superior performance compared to existing prompting approaches, achieving notable improvements, including 98% accuracy in Boolean expressions. Also, our ablation studies validate the critical contribution of each role, iterative refinement loops, and feedback mechanisms. Consequently, JoT significantly enhances accuracy, reliability, and consistency in binary reasoning tasks and shows potential for practical applications.

Problem

• Enhances binary logical reasoning in LLMs
• Addresses limitations in complex logical tasks
• Improves accuracy and reliability in reasoning

Innovation

• Multi-agent roles for logical reasoning
• Iterative refinement loops enhance accuracy
• Feedback mechanisms improve reliability