Beyond Tier Labels: Role- and Deployment-Dependent Model Substitution in Multi-Call LLM Workflows

2026-08-10

AI Summary

This work addresses a critical limitation in existing large language model (LLM) routing strategies, which rely solely on model-level labels to determine substitutions while neglecting the influence of a model’s role within multi-call workflows and its deployment context on actual performance. To remedy this, the authors propose a role-conditioned substitution principle that decouples replacement decisions into “whether to replace” and “effect evaluation” via predicate-action decomposition. Through a controlled solve-merge-validate pipeline, they systematically investigate the conditional dependencies governing effective substitutions. Empirical results from multi-call LLM workflow experiments—including input-matching interventions, allocation ablations, and cross-model comparisons across Qwen and GPT families—demonstrate that substitution efficacy is jointly determined by the model’s procedural role and deployment context. Notably, in mixed Qwen/GPT configurations, sparse, role-aware replacements reduce RMSE from 4.818 to 1.538, significantly outperforming indiscriminate full-model upgrades.

Abstract

Large multi-call LLM systems pose a scientific problem that query-level routing does not capture: the value of a model depends on where it enters a dependent computation and on the deployment that surrounds that call. Existing routers typically decide \emph{where} to spend a stronger model while treating the benefit of the substitution itself as known. We separate these two decisions through a predicate-action factorization and evaluate it in controlled solve-merge-verify workflows spanning 8-64 solve calls and four three-tier model ladders. The resulting evidence reveals a consistent principle beneath apparently conflicting outcomes. On numeric frequency counting, all-strong reduces RMSE from 4.818 to 1.538 in the Mixed Qwen/GPT ladder, whereas the average Qwen-only ordering reverses. Input-matched interventions further show that the same medium-to-strong action has sharply different value across roles and scales. A semantic task-and-contract shift reverses the Mixed ordering again, while allocation ablations distinguish useful sparse placement from under-coverage and indiscriminate escalation. Together, these results establish model substitution as a deployment-conditioned action rather than a property implied by a tier label, and they provide a practical sequence for large-scale workflow routing: calibrate the action, resolve its role-conditioned effect, and then optimize its placement.

Problem

• model substitution
• multi-call LLM workflows
• role-dependent effects
• deployment context
• tier labels

Innovation

• model substitution
• role-dependent routing
• deployment-conditioned action
• multi-call LLM workflows
• predicate-action factorization