IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows

2026-06-17

AI Summary

This work addresses the absence of evaluation frameworks for assessing voice agents’ ability to recover from interruptions during structured multi-step tasks. The authors introduce IHBench, the first benchmark designed to systematically evaluate how well voice agents handle user interjections, avoid redundant actions, and correctly resume task execution. Built upon a state-machine-driven workflow, IHBench injects six types of interruptions across ten enterprise domains and automatically scores performance along two dimensions—task completion and recovery quality—with human validation. Evaluation of 27 audio-language model configurations reveals that closed-source models significantly outperform open-source counterparts in task completion, robustness over long conversations, and cross-modal consistency. Furthermore, the proposed LLM-based scorer demonstrates strong alignment with human judgments.

Abstract

Voice agents deployed in structured workflows (customer service, healthcare scheduling, account management) must handle frequent user interruptions while maintaining progress through multi-step procedures. Existing benchmarks for speech-capable models focus on the timing of interruptions: barge-in detection, endpointing, and turn-taking dynamics. They leave unmeasured what happens after the interruption: does the agent resume the workflow at the correct step? Does it address the user's interjection? Does it avoid re-delivering content the user already heard? We introduce IHBench (Interruption Handling Benchmark), a benchmark that evaluates post-interruption recovery in voice agents executing state-machine-driven workflows across 10 enterprise domains. Six interruption types are injected at controlled points mid-utterance, with per-interruption evaluation rubrics generated alongside the data. Each interruption is scored on two axes: task fulfillment and recovery quality. We evaluate 27 audio-language model configurations from OpenAI, Google, and the open-weight community. Models vary widely, and recovery quality depends strongly on the interruption type. Across our experiments, closed-weight models are consistently more robust to interruptions than open-weight ones: they win far more often on task fulfillment, degrade roughly 3.3x more slowly as conversations grow longer, and show no audio-versus-text modality gap, whereas the open-weight models lose ground on all three. A human study validates the LLM judge against human annotators, and a cross-benchmark analysis against AudioMultiChallenge indicates that recovery quality is a largely distinct capability axis.

Problem

• voice agents
• interruption recovery
• structured workflows
• post-interruption evaluation
• task continuity

Innovation

• interruption handling
• voice agents
• post-interruption recovery
• structured workflows
• benchmarking