Measuring Pragmatic Influence in Large Language Model Instructions

2026-02-02

arXiv.org

AI Summary

研究通过构建框架测量大型语言模型指令中的语用影响，分解指令框架、分类策略并量化其对任务优先级的影响。

Abstract

It is not only what we ask large language models (LLMs) to do that matters, but also how we prompt. Phrases like"This is urgent"or"As your supervisor"can shift model behavior without altering task content. We study this effect as pragmatic framing, contextual cues that shape directive interpretation rather than task specification. While prior work exploits such cues for prompt optimization or probes them as security vulnerabilities, pragmatic framing itself has not been treated as a measurable property of instruction following. Measuring this influence systematically remains challenging, requiring controlled isolation of framing cues. We introduce a framework with three novel components: directive-framing decomposition separating framing context from task specification; a taxonomy organizing 400 instantiations of framing into 13 strategies across 4 mechanism clusters; and priority-based measurement that quantifies influence through observable shifts in directive prioritization. Across five LLMs of different families and sizes, influence mechanisms cause consistent and structured shifts in directive prioritization, moving models from baseline impartiality toward favoring the framed directive. This work establishes pragmatic framing as a measurable and predictable factor in instruction-following systems.

Problem

• pragmatic framing
• large language models
• instruction following
• contextual cues
• directive prioritization

Innovation

• pragmatic framing
• directive-framing decomposition
• taxonomy of framing strategies
• priority-based measurement