HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation

2026-07-05

AI Summary

This work addresses a critical gap in existing evaluation frameworks, which often overlook the active role of humans in large language model (LLM)-driven human-AI collaboration systems. The authors propose the HAS-Framework, which uniquely models both humans and LLM agents as first-class participants with explicit roles, permissions, and communication pathways. Complementing this, they introduce HAS-Bench, a configurable benchmark that supports diverse modes of human involvement. Collaboration dynamics are formalized through a graph-based representation, and system performance is assessed via multidimensional metrics—including clarification quality, feedback utilization, and control calibration—capturing both process-level interactions and outcome-level efficacy. Experiments across six domains demonstrate that strategically configuring the timing, modality, and role of human participation significantly enhances task completion rates and system robustness.

Abstract

Large language models increasingly operate in settings where humans are active collaborators rather than passive task providers. We introduce HAS-Framework, a graph-based framework that represents humans and LLM-powered agents as first-class participants with explicit roles, permissions, communication paths, and action authority. Building on this framework, HAS-Bench evaluates Human-Agent Systems under configurable human participation across agency levels, interaction channels, and persona policies. The benchmark measures both task outcomes and process-level collaboration behavior, including clarification quality, feedback utilization, control calibration, safety, initiative, and interaction cost. Experiments across six domains show that human participation can substantially improve task completion and failure recovery, but the gains depend on when, how, and by whom human input is exercised.

Problem

• Human-Agent Systems
• Large Language Models
• Human Participation
• Collaboration Evaluation
• Configurable Interaction

Innovation

• Human-Agent Collaboration
• Configurable Human Participation
• LLM-Based Systems
• Collaboration Benchmarking
• Graph-Based Framework