Sherlock: Reliable and Efficient Agentic Workflow Execution

2025-10-31

AI Summary

To address error propagation and consequent reliability degradation in LLM-based agent workflows, this paper proposes a selective verification framework. First, counterfactual analysis identifies error-prone nodes; second, a multi-strategy validator library—incorporating self-reflection, debate, and LLM-as-a-judge—is dynamically leveraged to select the optimal validator per node; third, speculative execution coupled with fault-aware rollback minimizes latency while preserving accuracy. Evaluated across multiple benchmarks, the framework achieves an average 18.3% accuracy gain, 48.7% speedup over non-speculative baselines, and reduces verification cost by 26.0% compared to Monte Carlo methods. Its core innovation lies in jointly modeling error sensitivity, adaptively selecting validators, and integrating lightweight execution optimizations—thereby achieving, for the first time, a synergistic balance between high reliability and low overhead in LLM agent systems.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

With the increasing adoption of large language models (LLM), agentic workflows, which compose multiple LLM calls with tools, retrieval, and reasoning steps, are increasingly replacing traditional applications. However, such workflows are inherently error-prone: incorrect or partially correct output at one step can propagate or even amplify through subsequent stages, compounding the impact on the final output. Recent work proposes integrating verifiers that validate LLM output or actions, such as self-reflection, debate, or LLM-as-a-judge mechanisms. Yet, verifying every step introduces significant latency and cost overheads. In this work, we seek to answer three key questions: which nodes in a workflow are most error-prone and thus deserve costly verification, how to select the most appropriate verifier for each node, and how to use verification with minimal impact to latency? Our solution, Sherlock, addresses these using counterfactual analysis on agentic workflows to identify error-prone nodes and selectively attaching cost-optimal verifiers only where necessary. At runtime, Sherlock speculatively executes downstream tasks to reduce latency overhead, while verification runs in the background. If verification fails, execution is rolled back to the last verified output. Compared to the non-verifying baseline, Sherlock delivers an 18.3% accuracy gain on average across benchmarks. Sherlock reduces workflow execution time by up to 48.7% over non-speculative execution and lowers verification cost by 26.0% compared to the Monte Carlo search-based method, demonstrating that principled, fault-aware verification effectively balances efficiency and reliability in agentic workflows.

Problem

• Identifying error-prone nodes in agentic workflows for targeted verification
• Selecting cost-optimal verifiers for specific workflow nodes
• Minimizing latency impact through speculative execution during verification

Innovation

• Uses counterfactual analysis to identify error-prone nodes
• Selectively attaches cost-optimal verifiers where necessary
• Speculatively executes downstream tasks to reduce latency