Human-in-the-Loop Annotation for Image-Based Engagement Estimation: Assessing the Impact of Model Reliability on Annotation Accuracy

2025-02-11

AI Summary

This study investigates how model reliability influences human trust, cognitive load, and behavioral consistency in human-in-the-loop (HITL) sentiment annotation. We designed a three-stage controlled experiment—baseline, model-error prompting, and negative framing—using image-based emotion recognition as the task. Data were collected via behavioral logs, validated cognitive load measures (e.g., NASA-TLX), and semi-structured interviews. Our findings reveal: (1) high model reliability significantly improves inter-annotator consistency (Cohen’s κ = 0.82); (2) explicit error prompts foster critical engagement but reduce consistency and increase perceived frustration; (3) negative framing paradoxically increases self-reported trust by +2.1 points (5-point scale), exposing a systematic cognitive bias in trust calibration. This work advances HITL research by empirically disentangling the joint effects of model reliability and psychological framing—challenging conventional HITL design assumptions—and provides both theoretical grounding and actionable guidelines for developing trustworthy, human-centered AI annotation systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Human-in-the-loop Machine Learning
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Human-in-the-loop (HITL) frameworks are increasingly recognized for their potential to improve annotation accuracy in emotion estimation systems by combining machine predictions with human expertise. This study focuses on integrating a high-performing image-based emotion model into a HITL annotation framework to evaluate the collaborative potential of human-machine interaction and identify the psychological and practical factors critical to successful collaboration. Specifically, we investigate how varying model reliability and cognitive framing influence human trust, cognitive load, and annotation behavior in HITL systems. We demonstrate that model reliability and psychological framing significantly impact annotators' trust, engagement, and consistency, offering insights into optimizing HITL frameworks. Through three experimental scenarios with 29 participants--baseline model reliability (S1), fabricated errors (S2), and cognitive bias introduced by negative framing (S3)--we analyzed behavioral and qualitative data. Reliable predictions in S1 yielded high trust and annotation consistency, while unreliable outputs in S2 led to increased critical evaluations but also heightened frustration and response variability. Negative framing in S3 revealed how cognitive bias influenced participants to perceive the model as more relatable and accurate, despite misinformation regarding its reliability. These findings highlight the importance of both reliable machine outputs and psychological factors in shaping effective human-machine collaboration. By leveraging the strengths of both human oversight and automated systems, this study establishes a scalable HITL framework for emotion annotation and lays the foundation for broader applications in adaptive learning and human-computer interaction.

Problem

• Model reliability impacts human trust
• Cognitive bias affects annotation accuracy
• Human-machine collaboration enhances emotion estimation

Innovation

• Human-in-the-Loop frameworks
• Model reliability evaluation
• Psychological impact analysis