Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection

2025-02-10

AI Summary

To address the problem that prompt strategies for social bias detection heavily depend on input context, while existing methods typically optimize only a single prompting technique and lack compositional adaptability, this paper proposes Input-aware Prompt Synthesis (IPrompt). IPrompt dynamically predicts the optimal combination of prompting techniques—e.g., chain-of-thought or role-playing—for each input instance, leveraging a lightweight prediction module to orchestrate multi-strategy prompting and jointly optimizing with instruction-tuned large language models. It introduces the first context-adaptive prompt composition selection mechanism, decoupling task specification, model architecture, and prompt design to significantly enhance generalization and robustness. Extensive evaluation across three LLMs and three bias benchmark datasets demonstrates consistent superiority over single-prompt baselines and state-of-the-art automatic prompting methods, achieving SOTA performance in most settings. Preliminary experiments further validate its cross-task transferability.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent advances on instruction fine-tuning have led to the development of various prompting techniques for large language models, such as explicit reasoning steps. However, the success of techniques depends on various parameters, such as the task, language model, and context provided. Finding an effective prompt is, therefore, often a trial-and-error process. Most existing approaches to automatic prompting aim to optimize individual techniques instead of compositions of techniques and their dependence on the input. To fill this gap, we propose an adaptive prompting approach that predicts the optimal prompt composition ad-hoc for a given input. We apply our approach to social bias detection, a highly context-dependent task that requires semantic understanding. We evaluate it with three large language models on three datasets, comparing compositions to individual techniques and other baselines. The results underline the importance of finding an effective prompt composition. Our approach robustly ensures high detection performance, and is best in several settings. Moreover, first experiments on other tasks support its generalizability.

Problem

• Optimize prompt composition adaptively
• Enhance social bias detection accuracy
• Ensure high performance across contexts

Innovation

• Adaptive prompting optimizes composition dynamically
• Technique predicts optimal prompts ad-hoc
• Ensures robust social bias detection performance