Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy

2025-07-04

AI Summary

To address the inefficiency and human-intensive trial-and-error nature of conventional prompt engineering, this paper proposes a declarative prompt optimization method built on the DSPy framework, jointly optimizing instruction tuning and in-context example selection. We systematically evaluate the approach across five diverse tasks—gatekeeping rules, hallucination detection, code generation, and two additional domains—demonstrating substantial improvements: prompt evaluation accuracy increases by 17.8 percentage points (from 46.2% to 64.0%), and routing agent accuracy reaches 90.0% (+5.0%). Crucially, optimized prompts enable smaller language models to approach the performance of stronger baselines, empirically validating the “prompt-as-code” paradigm. This work constitutes the first multi-scenario empirical study confirming DSPy’s significant, task-dependent performance gains for LLMs, establishing a reproducible methodology and evidence base for automated prompt engineering.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Although prompt engineering is central to unlocking the full potential of Large Language Models (LLMs), crafting effective prompts remains a time-consuming trial-and-error process that relies on human intuition. This study investigates Declarative Self-improving Python (DSPy), an optimization framework that programmatically creates and refines prompts, applied to five use cases: guardrail enforcement, hallucination detection in code, code generation, routing agents, and prompt evaluation. Each use case explores how prompt optimization via DSPy influences performance. While some cases demonstrated modest improvements - such as minor gains in the guardrails use case and selective enhancements in hallucination detection - others showed notable benefits. The prompt evaluation criterion task demonstrated a substantial performance increase, rising accuracy from 46.2% to 64.0%. In the router agent case, the possibility of improving a poorly performing prompt and of a smaller model matching a stronger one through optimized prompting was explored. Although prompt refinement increased accuracy from 85.0% to 90.0%, using the optimized prompt with a cheaper model did not improve performance. Overall, this study's findings suggest that DSPy's systematic prompt optimization can enhance LLM performance, particularly when instruction tuning and example selection are optimized together. However, the impact varies by task, highlighting the importance of evaluating specific use cases in prompt optimization research.

Problem

• Optimizing prompts programmatically for better LLM performance
• Evaluating DSPy in diverse use cases like code generation
• Assessing task-specific impacts of systematic prompt refinement

Innovation

• DSPy programmatically creates and refines prompts
• Optimizes instruction tuning and example selection
• Enhances LLM performance across varied use cases