Better Together, in the Right Order: Classical-then-LLM Optimization for SE

2026-06-30

AI Summary

This work addresses the challenge of balancing efficiency and accuracy when applying large language models (LLMs) to software engineering configuration tasks. To this end, we propose SNAP2, a two-stage collaborative optimization paradigm that first employs a low-cost classical optimizer for coarse-grained search to generate an initial solution, followed by fine-grained refinement using an LLM, thereby synergistically combining the strengths of both approaches. Evaluated across over 100 software engineering tasks, SNAP2 achieves state-of-the-art performance in 85% of cases—significantly outperforming pure LLM-based methods (75%)—while reducing token consumption by 30% and accelerating execution by 1.4×. These results overturn the conventional integration paradigm dominated solely by classical optimizers.

Abstract

A growing body of work combines large language models (LLMs) with classical optimizers for software engineering (SE) configuration tasks. Often, the classical optimizer is in charge: it owns the search loop and calls the LLM only to assist in subroutines (e.g. to warm-start the first generation, propose a mutation, or stand in as a surrogate). We report that there is much value in the reverse approach: seeding an LLM with the results from a cheap classical learner. We call this method SNAP2. Applied to over 100 SE tasks, it is the single best of all methods studied, reaching the top tier on 85% of tasks, ahead of the same LLM run alone (75%) and ahead of every method in which the classical optimizer retains control. It is also less expensive: relative to the LLM-alone method, it uses roughly 30% fewer tokens and runs 1.4x faster, since the classical setup performs the inexpensive work, and the LLM is invoked only to finish. We conclude that it is unwise to study classical learners or LLMs in isolation: there is much value in combining the two, and in the order that combination is applied.

Problem

• classical optimization
• large language models
• software engineering
• configuration tasks
• hybrid methods

Innovation

• classical-then-LLM
• SNAP2
• software engineering optimization
• hybrid optimization
• large language models