Open Problems in Constitutional Preference Reconstruction

2026-06-29

AI Summary

Existing constitutional preference elicitation methods represent principles as flat lists, lacking compositional logic, which renders decision rules unexecutable and inadequately evaluated. This work systematically investigates the performance of constitutional approaches on three open challenges: principle quality metrics, compositional ambiguity, and inter-model variability. It empirically demonstrates for the first time that constitutions must be evaluated as integrated “constitution–executor” systems. To address these limitations, the paper introduces ICAI+, a principled refinement framework. Experiments on PRISM, AlpacaEval, and Chatbot Arena datasets—employing executors such as Inverse Constitutional AI (ICAI), LLM judges, and majority voting—show that ICAI+ improves agreement across executors from 73% to 78%, while achieving comparable accuracy between transparent executors (66%) and LLM judges (67%).

Abstract

Pairwise preference data is widely used for training and evaluating language models (e.g., RLHF), but each datapoint records a \emph{choice}, not the rationale behind it. Methods such as Inverse Constitutional AI (ICAI) attempt to improve interpretability by compressing datasets into short ``constitutions'' of natural-language principles. We argue this framing is under-specified: a flat list of principles is not yet an executable decision rule because it leaves principle composition implicit. We use the pairwise setting as a testbed to empirically characterize three open problems in constitutional methods. First, principle quality is hard to measure: coverage and accuracy are useful but incomplete proxies for end-to-end reconstruction. Second, \emph{composition is ambiguous}: holding principles fixed, different executors (LLM judge versus majority vote) agree only $73\%$ of the time. Third, \emph{constitutions differ between LLMs}: cross-model vote agreement is $73\%$, whereas intra-model agreement is $81\%$. Across PRISM, AlpacaEval, and Chatbot Arena, we show that principle refinement (ICAI+) may be a first step towards ameliorating these problems: inter-executor agreement rises to $78\%$, and transparent executors match LLM judge accuracy ($66\%$ vs.\ $67\%$). Our results highlight that constitutions should be evaluated as \emph{constitution--executor systems}, with implications for LLMs-as-a-judge broadly.

Problem

• constitutional preference reconstruction
• principle composition
• executor agreement
• LLM-as-a-judge
• preference data interpretability

Innovation

• Constitutional AI
• Preference Reconstruction
• Principle Composition
• LLM-as-a-Judge
• Interpretability