PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data

2025-12-11

AI Summary

Manual design of high-quality prompts is challenging in few-shot settings, while existing automated methods suffer from low efficiency and reliance on human-crafted demonstrations. Method: We propose ShapleyPrompt—the first framework to incorporate Monte Carlo Shapley values into in-context learning prompt construction. It quantifies the marginal contribution of each candidate example to model performance, enabling efficient, differentiable-budget-aware example selection and dynamic editing (addition, deletion, or retention). Our approach integrates aggressive sub-sampling, replay buffering, and LLM-based zero-/few-shot evaluation—eliminating the need for human-authored examples. Results: ShapleyPrompt significantly outperforms state-of-the-art automated prompting methods on text simplification, GSM8K, and multi-class classification. With increased computational budget, it establishes new SOTAs across all three tasks, demonstrating superior data efficiency and example quality.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Active Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

LLMs are highly sensitive to prompt design, but handcrafting effective prompts is difficult and often requires intricate crafting of few-shot examples. We propose a fast automatic prompt construction algorithm that augments human instructions by generating a small set of few shot examples. Our method iteratively replaces/drops/keeps few-shot examples using Monte Carlo Shapley estimation of example utility. For faster execution, we use aggressive subsampling and a replay buffer for faster evaluations. Our method can be run using different compute time budgets. On a limited budget, we outperform existing automatic prompting methods on text simplification and GSM8K and obtain second best results on classification and summarization. With an extended, but still modest compute budget we set a new state of the art among automatic prompting methods on classification, simplification and GSM8K. Our results show that carefully constructed examples, rather than exhaustive instruction search, are the dominant lever for fast and data efficient prompt engineering. Our code is available at https://github.com/Batorskq/PIAST.

Problem

• Addresses LLMs' sensitivity to prompt design by automating construction.
• Generates few-shot examples to augment human instructions efficiently.
• Optimizes example selection using Monte Carlo Shapley estimation.

Innovation

• Automatic prompt construction algorithm with few-shot examples
• Iterative example selection using Monte Carlo Shapley estimation
• Aggressive subsampling and replay buffer for faster execution