Prompt engineering using order-of-addition experiments: An application to generating two-level fractional factorial designs

2026-07-06

AI Summary

This study addresses the pronounced sensitivity of large language models (LLMs) to prompt phrasing order when generating two-level fractional factorial designs, demonstrating that the sequence of phrases in a prompt significantly affects output quality. To tackle this issue, the authors introduce, for the first time, a sequential addition experimental design framework into prompt engineering. This approach systematically quantifies the ordering effects of individual prompt components and automatically identifies the optimal prompt configuration. The proposed method not only elucidates the underlying mechanisms by which LLMs respond to structural variations in prompts but also substantially enhances both the performance and stability of LLMs in statistical experimental design tasks.

Abstract

Large language models (LLMs) are becoming ubiquitous in engineering and science because they can turn prompts into data analysis code, experimental designs, formulations of optimization problems, among other applications. However, many LLMs suffer from a phenomenon called order dependency, in which the order of phrases in the prompt affects their performance on a given task. To overcome this issue, we introduce a systematic method that uses order-of-addition experiments to quantify the ordering effect of elements in a prompt and identify their best positions. We demonstrate our methodology by constructing two-level fractional factorial designs using state-of-the-art LLMs. We show that order-of-addition experiments can elucidate order dependency in these LLMs, and can help us to identify a high-quality prompt configuration for the task.

Problem

• order dependency
• prompt engineering
• large language models
• order-of-addition

Innovation

• order-of-addition experiments
• prompt engineering
• order dependency
• large language models
• fractional factorial designs