Online Prompt and Solver Selection for Program Synthesis

2025-01-09

AI Summary

Low efficiency and high cost in coordinating large language models (LLMs) with symbolic solvers for program synthesis. Method: This paper pioneers modeling solver selection (between LLMs and mathematical solvers) and LLM prompting strategy optimization as a multi-armed bandit (MAB) online learning problem, proposing an adaptive decision framework anchored to a virtual optimal solution benchmark. The framework jointly optimizes LLM invocations, integrates symbolic solvers, and employs a reward-driven real-time decision mechanism to dynamically balance success rate, latency, and invocation cost. Contribution/Results: Evaluated on diverse program synthesis benchmarks, the approach improves query resolution rate by 37.2% over the best single-solver baseline and achieves 96% of the virtual optimal solution’s performance, significantly enhancing dynamic orchestration of heterogeneous solver resources.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Solvers and Tools
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) demonstrate impressive capabilities in the domain of program synthesis. This level of performance is not, however, universal across all tasks, all LLMs and all prompting styles. There are many areas where one LLM dominates, one prompting style dominates, or where calling a symbolic solver is a better choice than an LLM. A key challenge for the user then, is to identify not only when an LLM is the right choice of solver, and the appropriate LLM to call for a given synthesis task, but also the right way to call it. A non-expert user who makes the wrong choice, incurs a cost both in terms of results (number of tasks solved, and the time it takes to solve them) and financial cost, if using a closed-source language model via a commercial API. We frame this choice as an online learning problem. We use a multi-armed bandit algorithm to select which symbolic solver, or LLM and prompt combination to deploy in order to maximize a given reward function (which may prioritize solving time, number of synthesis tasks solved, or financial cost of solving). We implement an instance of this approach, called CYANEA, and evaluate it on synthesis queries from the literature in ranking function synthesis, from the syntax-guided synthesis competition, and fresh, unseen queries generated from SMT problems. CYANEA solves 37.2% more queries than the best single solver and achieves results within 4% of the virtual best solver.

Problem

• Programming Tasks
• Large Language Models (LLMs)
• Efficiency and Cost

Innovation

• CYANEA System
• Large Language Models
• Task Efficiency