Democratic ICAI: Debating Our Way to Steering Principles from Preferences

2026-06-26

AI Summary

This work addresses the limitations of existing preference alignment methods, which struggle to capture the multidimensional reasoning underlying human judgments and rely solely on pairwise labels, thereby constraining interpretability and expressiveness. To overcome these challenges, the authors propose a structured role-based debate framework for preference modeling: multiple perspectives engage in competitive debate to generate rationales, which are then distilled into natural language guiding principles. These principles are integrated with large language model prompting and decision trees to predict preferences. The approach substantially enhances both the expressiveness of the derived principles and the interpretability of decisions. Evaluated on the MuCE-Pref and LiTBench benchmarks, the method outperforms current baselines in preference prediction accuracy, and its generated principles are consistently preferred by human evaluators.

Abstract

Preference-based alignment often struggles to capture the reasoning that underlies human judgments. Many evaluations rely on multiple interacting criteria, yet pairwise labels reveal only the final choice rather than the considerations that shape preferences. Inverse Constitutional AI (ICAI) improves interpretability in decision making by summarizing preferences into natural-language principles, but its single-pass explanations miss much of the nuance involved in complex decisions. We introduce Democratic ICAI, a novel approach that gathers multiple competing rationales through structured persona debate, offering a broader and more expressive account of the factors influencing each comparison. From these richer signals, we derive clearer and more comprehensive steering principles and use them to guide decision modeling through both LLM-based and decision-tree judges. Experiments on creative preference benchmarks, MuCE-Pref and LiTBench, across multiple creative task categories show that Democratic ICAI yields a more faithful preference structure. It improves average preference prediction across tasks relative to deliberative prompting and principle-based baselines, while producing constitutions that LLM annotators prefer.

Problem

• preference-based alignment
• interpretability
• complex decision-making
• human judgment reasoning
• steering principles

Innovation

• Democratic ICAI
• preference alignment
• persona debate
• steering principles
• interpretability