Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering

2025-07-18

AI Summary

In high-stakes scenarios, reasoning-intensive large language models (LLMs) suffer from high sporadic error rates and substantial response latency. Method: We propose a human-AI collaborative “fail-fast-or-ask” mechanism: when model uncertainty is high—proxied by reasoning-chain length—the system automatically escalates to human experts; concurrently, a lightweight non-reasoning model pre-filters simple inputs to alleviate the “latency drag” bottleneck. Our approach follows a black-box systems engineering paradigm, integrating uncertainty estimation, dynamic task routing, and human-AI coordination policies. Results: Experiments show that, while maintaining >90% area under the accuracy–rejection-rate curve, the system reduces end-to-end latency by ~40% and cuts computational cost by 50%, significantly enhancing both reliability and practicality of AI reasoning in safety-critical applications.

Technology Category

• Reasoning under Uncertainty: Applications
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

State-of-the-art reasoning LLMs are powerful problem solvers, but they still occasionally make mistakes. However, adopting AI models in risk-sensitive domains often requires error rates near 0%. To address this gap, we propose collaboration between a reasoning model and a human expert who resolves queries the model cannot confidently answer. We find that quantifying the uncertainty of a reasoning model through the length of its reasoning trace yields an effective basis for deferral to a human, e.g., cutting the error rate of Qwen3 235B-A22B on difficult MATH problems from 3% to less than 1% when deferring 7.5% of queries. However, the high latency of reasoning models still makes them challenging to deploy on use cases with high query volume. To address this challenge, we explore fronting a reasoning model with a large non-reasoning model. We call this modified human-in-the-loop system "Fail Fast, or Ask", since the non-reasoning model may defer difficult queries to the human expert directly ("failing fast"), without incurring the reasoning model's higher latency. We show that this approach yields around 40% latency reduction and about 50% cost savings for DeepSeek R1 while maintaining 90+% area under the accuracy-rejection curve. However, we observe that latency savings are lower than expected because of "latency drag", the phenomenon that processing easier queries with a non-reasoning model pushes the reasoning model's latency distribution towards longer latencies. Broadly, our results suggest that the deficiencies of state-of-the-art reasoning models -- nontrivial error rates and high latency -- can be substantially mitigated through black-box systems engineering, without requiring access to LLM internals.

Problem

• Reducing error rates in reasoning LLMs for risk-sensitive domains
• Decreasing high latency in reasoning models for high query volume
• Mitigating reasoning model deficiencies via human-in-the-loop systems

Innovation

• Human-in-the-loop resolves uncertain model queries
• Non-reasoning model defers queries to reduce latency
• Quantifying uncertainty via reasoning trace length