Evaluating Human-AI Collaboration: A Review and Methodological Framework

2024-07-09

arXiv.org

AI Summary

Current human-AI collaboration (HAIC) evaluation lacks a unified framework capable of accommodating the heterogeneity and dynamic reciprocity inherent across AI-centered, human-centered, and symbiotic HAIC paradigms. Method: This paper proposes the first structured evaluation framework tailored to all three HAIC modes, introducing a novel multi-dimensional evaluation decision tree that integrates quantitative and qualitative metrics—combining objective and subjective dimensions—and formally modeling dynamic reciprocity throughout the collaborative process. The framework is validated empirically across four domains—manufacturing, healthcare, finance, and education—via systematic literature review and cross-domain adaptation. Contribution/Results: Results demonstrate significant improvements in evaluation specificity, interpretability, and practical guidance value. The framework establishes a methodological foundation for scientifically measuring HAIC effectiveness, enabling rigorous, context-sensitive assessment of collaborative outcomes.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved. This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems. Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.

Problem

• Challenges in evaluating Human-AI Collaboration effectiveness.
• Need for a structured framework to assess HAIC systems.
• Application of evaluation metrics across diverse domains.

Innovation

• Structured decision tree for metric selection
• Combines quantitative and qualitative evaluation metrics
• Applicable across diverse domains like healthcare, finance