BT-APE: A Computationally Light Backtracking Approach to Automatic Prompt Engineering for Requirements Classification

2026-07-01

AI Summary

This work addresses the high computational cost of existing automated prompt engineering methods and the inconsistent performance of handcrafted prompts in intent classification tasks. We propose BT-APE, a lightweight framework that formulates prompt design as an optimization problem, leveraging large language models to generate candidate prompts and integrating backtracking search with dynamic example selection to iteratively refine classification performance. Systematic evaluations on three benchmark datasets demonstrate that BT-APE achieves accuracy comparable to the state-of-the-art method PE2 while reducing input tokens by approximately 72% and runtime by 66%, substantially outperforming conventional baselines. The source code, an interactive tool, and a reproducibility package are publicly released.

Abstract

Large language models (LLMs) are increasingly applied to requirements engineering (RE) tasks, yet the prompts guiding them are typically designed manually through trial and error, yielding inconsistent and suboptimal results. Automated prompt construction remains largely unexplored in RE, leaving its effectiveness unclear. To address this, we propose a lightweight Automatic Prompt Engineering approach, Backtracking APE (BT-APE), and apply it to requirements classification. We frame prompt design as an optimization problem, iteratively refining prompts via LLM-generated candidates, backtracking search, and dynamic example selection. Evaluating BT-APE on three benchmark datasets with five instruction-tuned LLMs, we compare it against four classical prompting baselines (zero-shot, few-shot, chain-of-thought, CoT+few-shot) and a state-of-the-art but resource-intensive APE baseline (PE2). BT-APE and PE2 achieve nearly identical accuracy, both substantially outperforming the classical baselines with large effect sizes; however, BT-APE imposes a far lighter computational footprint, consuming roughly 72% fewer input tokens and 66% less wall-clock time at equivalent accuracy, making it better suited to resource-constrained deployment. Our contributions are threefold: (i) a lightweight APE framework with an open interactive tool and replication package; (ii) the first systematic comparison of APE against classical prompting for requirements classification; and (iii) insights into how class definitions and prompt evolution affect performance.

Problem

• Automatic Prompt Engineering
• Requirements Classification
• Large Language Models
• Prompt Optimization
• Requirements Engineering

Innovation

• Automatic Prompt Engineering
• Backtracking Search
• Requirements Classification
• Lightweight Optimization
• LLM Prompting