ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

2026-06-23

AI Summary

This work addresses limitations in existing hyperparameter optimization (HPO) methods, which are constrained by the inductive bias of a single optimizer and, in the case of large language model (LLM)-based approaches, focus solely on iteration count while neglecting end-to-end wall-clock time. The authors propose ASAP, a co-designed agent-system framework that integrates multiple biased optimizers on the agent side, with an LLM dynamically selecting recommendations, and restructures the execution loop on the system side through prefix-stable prompting, speculative parallelism, KV cache reuse, and adaptive tuning. ASAP is the first to unify tool integration with end-to-end time optimization, achieving both low regret and high efficiency across diverse HPO tasks, significantly outperforming current baselines.

Abstract

Hyperparameter Optimization (HPO) is essential for maximizing machine learning model performance, and its core challenge is sample efficiency: finding strong configurations within a limited budget. Because every HPO tool relies on a surrogate prior that imparts its own inductive bias, individual tools struggle once problems become sufficiently diverse and drift from these priors. Motivated by the reasoning and generalization capabilities of LLMs, recent work has explored using LLMs for HPO and reports improved per-iteration performance. Yet these methods share two limitations with a common origin: they use the LLM as a single-tool replacement evaluated by iteration count. (i) Deployed in place of prior tools, the LLM is itself constrained by its pretraining objective to one family of inductive-biased proposals; this single-source setup still fails to handle the full diversity of problems. (ii) Per-iteration evaluation ignores that, in real runs, LLM inference or tool execution is paid serially on top of model evaluation every round, so iteration-count gains do not translate into end-to-end wall-clock gains. We present ASAP, an agent-system co-design that addresses both limitations. On the agent side, ASAP uses the LLM to integrate a diverse pool of inductive-biased optimizers and to select among their proposals each round. On the system side, ASAP re-architects the loop to reduce end-to-end wall-clock while preserving regret quality: a prefix-stable prompt maximizes KV-cache reuse across rounds; speculation parallelism hides the remaining LLM and tool latency under model evaluation via a relative-error accept test; and a Self-Tuner adapts the speculation threshold from execution logs off the critical path. Extensive experiments on diverse modern HPO tasks show that ASAP consistently outperforms baselines, underscoring the value of tool integration and agent-system co-design.

Problem

• Hyperparameter Optimization
• Large Language Models
• Inductive Bias
• Wall-Clock Time
• Sample Efficiency

Innovation

• agent-system co-design
• hyperparameter optimization
• large language models
• wall-clock efficiency
• speculation parallelism