Understanding the Challenges in Iterative Generative Optimization with LLMs

2026-03-25

AI Summary

This work systematically investigates three critical yet often overlooked design factors that profoundly influence the effectiveness of iterative generative optimization with large language models: the choice of initial artifacts, the scope of credit assignment in execution trajectories, and the batching strategy for trial-and-error samples. Through extensive experiments across diverse benchmarks—including MLAgentBench, Atari, and BigBench Hard—combined with execution feedback and iterative editing mechanisms, the study empirically demonstrates that these “hidden” choices decisively determine optimization success or failure. Specifically, different initial artifacts significantly affect the reachability of the solution space, truncated trajectories can still enhance performance on Atari tasks, and increasing batch size does not necessarily improve generalization. These findings provide both theoretical grounding and practical guidance for building robust iterative self-improvement systems.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make ``hidden'' design choices: What can the optimizer edit and what is the "right" learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices.

Problem

• generative optimization
• large language models
• iterative improvement
• learning loop
• design choices

Innovation

• generative optimization
• large language models
• iterative improvement
• learning loop design
• execution feedback