Maestro Order: A Model-Agnostic Orchestration Harness

2026-06-22

AI Summary

While single-pass forward inference models are efficient, they often produce unreliable outputs due to high-confidence hallucinations. To address this, this work proposes a model-agnostic orchestration framework that dynamically composes arbitrary black-box models into a reliable problem-solving system through four structural primitives—decomposition, integration, verification, and recursion—guided by a budget-aware controller. The framework uniquely integrates structured composition with online verification and incorporates a cost-aware strategy to maximize reliability gain per unit of computational cost while significantly reducing overall overhead. Experimental results demonstrate that a two-stage verification gate improves reliability from 0.55 to 0.98, and a four-stage configuration achieves 0.999; furthermore, the budget-aware controller attains target reliability levels at substantially lower costs than majority voting baselines.

Abstract

A single forward pass of a capable model is a fast, fluent, and unreliable problem-solver: it is right often enough to be useful and wrong often enough to be dangerous; in language models, such confident errors are known as hallucinations. We present Maestro Order, a model-agnostic orchestration harness that turns unreliable solvers into reliable problem-solving systems by composing them according to four structural primitives (decompose, ensemble, verify, and recurse) and a budget-aware controller that decides where to spend compute. The harness treats any model as a black-box base solver behind a uniform interface, layers a verifier ensemble whose discrimination is measured online, and allocates verification and voting to the stages with the highest marginal reliability per unit cost. We give the architecture, the message and state schema, the controller algorithm, and the engineering that makes it deterministic, observable, and fault-tolerant. We then specify an evaluation methodology (reliability at fixed cost, coverage, calibration, and ablations) and report results from a faithful Monte Carlo simulation of the harness over a parameterized solver/verifier model. The simulation reproduces the predicted laws quantitatively: verification amplifies reliability geometrically (e.g. $0.55\to0.98$ with two gates, $\to0.999$ with four), voting helps only above chance and is limited by shared errors, and a budget-aware controller reaches a target reliability at a small fraction of the cost of voting alone by selecting the cheapest mechanism for each regime. We close with failure modes (verifier gaming, correlated errors, and decomposition error compounding) and concrete guidance: build robust checkers, diversify solvers, and let the controller put compute where the information is.

Problem

• hallucination
• reliability
• model-agnostic
• verification
• compute budget

Innovation

• model-agnostic orchestration
• verification ensemble
• budget-aware control
• reliability amplification
• hallucination mitigation