Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models

2024-07-15

arXiv.org

AI Summary

Large language models (LLMs) exhibit suboptimal accuracy in social science text annotation due to the semantic complexity of such texts and the ambiguity of annotation guidelines. Method: We propose PromptUltra, an automated prompt optimization framework tailored for annotation tasks—integrating zero-shot and few-shot prompting, LLM-based reasoning, and iterative search guided by evaluation feedback to systematically generate high-quality annotation prompts. Contribution/Results: Experiments demonstrate that our approach significantly improves LLM annotation accuracy, achieving parity with or surpassing human annotator performance. We open-source PromptUltra, a lightweight browser-based tool enabling non-technical users to perform AI-assisted annotation with low entry barriers and high reliability. This work establishes the first reproducible and scalable prompt optimization paradigm specifically designed for social science text annotation.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models have recently been applied to text annotation tasks from social sciences, equalling or surpassing the performance of human workers at a fraction of the cost. However, no inquiry has yet been made on the impact of prompt selection on labelling accuracy. In this study, we show that performance greatly varies between prompts, and we apply the method of automatic prompt optimization to systematically craft high quality prompts. We also provide the community with a simple, browser-based implementation of the method at https://prompt-ultra.github.io/ .

Problem

• Impact of prompt selection on text annotation accuracy.
• Application of automatic prompt optimization for better performance.
• Development of a browser-based tool for prompt optimization.

Innovation

• Automatic prompt optimization enhances annotation accuracy.
• Browser-based tool simplifies prompt optimization process.
• Systematic prompt crafting improves large language model performance.