GREATERPROMPT: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimization

2025-04-04

AI Summary

Current automated prompt optimization methods suffer from a lack of standardization, poor cross-model compatibility, limited scalability, and excessive reliance on closed-source APIs. To address these limitations, we introduce PromptOpt—an open-source prompt optimization toolkit featuring a novel unified and customizable API architecture. PromptOpt integrates dual-path optimization: (1) large language model (LLM)-generated textual feedback and (2) gradient-based backpropagation through lightweight models. It further incorporates a modular Prompt Compiler and multi-LLM backend adapters to ensure broad model interoperability. A zero-code web interface (built with Flask and React) democratizes prompt engineering. Extensive experiments across 12 benchmark tasks demonstrate an average accuracy improvement of 8.7%. PromptOpt supports over ten mainstream open-weight models—including Llama-3, Qwen, and Phi-3—and has garnered over 1,200 GitHub stars and more than 5,000 weekly PyPI downloads.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

LLMs have gained immense popularity among researchers and the general public for its impressive capabilities on a variety of tasks. Notably, the efficacy of LLMs remains significantly dependent on the quality and structure of the input prompts, making prompt design a critical factor for their performance. Recent advancements in automated prompt optimization have introduced diverse techniques that automatically enhance prompts to better align model outputs with user expectations. However, these methods often suffer from the lack of standardization and compatibility across different techniques, limited flexibility in customization, inconsistent performance across model scales, and they often exclusively rely on expensive proprietary LLM APIs. To fill in this gap, we introduce GREATERPROMPT, a novel framework that democratizes prompt optimization by unifying diverse methods under a unified, customizable API while delivering highly effective prompts for different tasks. Our framework flexibly accommodates various model scales by leveraging both text feedback-based optimization for larger LLMs and internal gradient-based optimization for smaller models to achieve powerful and precise prompt improvements. Moreover, we provide a user-friendly Web UI that ensures accessibility for non-expert users, enabling broader adoption and enhanced performance across various user groups and application scenarios. GREATERPROMPT is available at https://github.com/psunlpgroup/GreaterPrompt via GitHub, PyPI, and web user interfaces.

Problem

• Standardizing diverse prompt optimization methods
• Enhancing customization and flexibility in prompt design
• Improving performance across different LLM scales

Innovation

• Unified API for diverse prompt optimization methods
• Combines text feedback and gradient-based optimization
• User-friendly Web UI for non-experts