FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect

2026-09-07

AI Summary

研究通过FramingQA基准测试，评估大型语言模型对问题表述的敏感度及其在不同领域内提供一致答案的能力。

Abstract

We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations. Large language models (LLMs) often change their responses to subtle rephrasings that align with an implied stance by users. This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts, and the consequences are highly costly in high-stakes domains. Because in the realistic scenarios, both expert practitioners and non-expert users frequently ask LLMs questions containing incomplete or misleading assumptions, models are highly susceptible to those framings. To test this, we inject the framing bias across three nested levels: a framing-biased question phrasing (root), an injected framing-biased premise prepended to a neutral question (propositional), and a premise paired with a framing-biased question (global). Evaluating nine open models (3.8B-70B) across four families, we find that strong per-variant accuracy does not guarantee the robustness across differently phrased questions under the fixed factual information.

Problem

• question framing
• large language models
• sensitivity
• high-stakes domains
• factual information

Innovation

• FramingQA
• Question Framing
• Model Sensitivity
• Large Language Models
• Benchmark