human-in-the-loop

To address the low efficiency of knowledge structuring in scientific literature—exacerbated by its exponential growth and heavy reliance on expert curation—this paper proposes a neural-symbolic, human-in-the-loop (HITL) workflow. The method leverages large language models (LLMs) to automatically extract and structure scholarly information, which is then ingested into the Open Research Knowledge Graph (ORKG). A modular architecture enables customizable LLM selection and multi-stage human verification, tightly coupling automation with expert oversight. Its key innovation lies in pioneering a synergistic mechanism between LLMs and symbolic knowledge graphs within the HITL paradigm. Evaluation shows the system achieves a System Usability Scale (SUS) score of 84.17 (A+ level), and reduces per-paper knowledge modeling time from hours to weeks down to an average of 24 minutes and 40 seconds—demonstrating substantial gains in scientific knowledge transformation efficiency.

This study addresses suboptimal personalization and low knowledge retention in STEM education by proposing a human-in-the-loop (HITL)-driven generative AI adaptive learning framework. Methodologically, it integrates retrieval-augmented generation (RAG), structured prompt engineering, and a student feedback labeling mechanism, enabling learners to critically annotate and collaboratively refine AI-generated content. Feedback labels are used to train a dynamic response optimization model that supports student-driven, iterative AI refinement. The key contribution lies in explicitly modeling fine-grained human feedback as computable signals and embedding this closed-loop feedback directly into the generative pipeline. Preliminary experiments demonstrate statistically significant improvements: +23.6% in knowledge retention, +31.2% in classroom engagement, and enhanced self-efficacy (p < 0.01), validating the efficacy of feedback-driven human–AI collaboration for personalized STEM instruction.

High annotation costs and prolonged turnaround times plague NLP development, necessitating efficient and reliable data labeling paradigms. This paper proposes an LLM-powered Human-in-the-Loop (HITL) hybrid annotation framework that systematically integrates synthetic data generation, active learning, and human-AI collaboration, augmented with built-in mechanisms for annotation quality assessment, annotator management, and cost-benefit analysis. Unlike prior work—largely theoretical or narrowly scoped—this study introduces the first deployable, plug-and-play industrial-grade annotation methodology, bridging the critical gap between methodological research and real-world engineering practice. Empirical validation across multiple production NLP projects demonstrates that the framework consistently reduces annotation costs and cycle time by 30–50%, while maintaining label quality within required thresholds.

This work addresses the limitations of current automatic text simplification approaches, which overly rely on automated metrics that fail to capture users’ actual comprehension abilities and normative standards, thereby offering inadequate support for cognitive accessibility. To overcome this, the authors propose a human-in-the-loop hybrid framework that integrates real-time human guidance during large language model generation alongside post-hoc human oversight. For the first time, this framework systematically embeds human roles into both generation and evaluation phases, leveraging a standards-aligned checklist, an Event-Condition-Action (ECA) rule engine, and accessibility-oriented key performance indicators (KPIs) to enable a traceable, reproducible, and auditable text generation pipeline. Empirical results demonstrate that the approach effectively encodes human feedback, enhances model adaptability, and provides a structured, transparent, and inclusive pathway for evaluating and optimizing accessible texts.

This study investigates whether large language models (LLMs) can reliably replace human annotators to mitigate the high cost and logistical complexity of human-subject studies in software engineering innovation evaluation. We systematically evaluate six state-of-the-art LLMs across ten code-related annotation tasks—including code summary quality assessment and defect repair judgment—using five public datasets. Methodologically, we propose *inter-model agreement* as a novel task-adaptability predictor and integrate confidence-threshold filtering to identify samples safe for LLM-only annotation, thereby establishing a hybrid human–LLM evaluation paradigm. Results show that LLMs achieve or approach human inter-annotator agreement (Krippendorff’s α ≥ 0.8) on multiple tasks; inter-model agreement strongly predicts task feasibility (AUC = 0.92); and confidence-based filtering raises replacement accuracy to 94.3%.

This work addresses the limitations of existing human-in-the-loop (HITL) mechanisms in intelligent agent workflows, which are often tightly coupled with application logic, resulting in poor reusability, weak consistency, and limited scalability. To overcome these challenges, the paper proposes a decoupled HITL system architecture that abstracts human oversight into an independent component. By introducing explicit interfaces and a structured execution model, the approach cleanly separates human–machine interaction from business logic. Furthermore, it introduces a novel four-dimensional framework—comprising intervention conditions, role resolution, interaction semantics, and communication channels—to enable context-aware, controllable human intervention. This design achieves, for the first time, protocol-level reusability of HITL mechanisms, supporting consistent and scalable autonomy governance in multi-agent environments and laying a foundational infrastructure for system-level human–agent collaboration.

This study addresses the lack of a clearly defined normative role for human annotators in current reinforcement learning from human feedback (RLHF) approaches, which leads to ambiguous annotation protocols and risks of model failure. The work systematically introduces three normative role models—expansion, evidence, and authority—and argues that annotation processes should be decomposed along task dimensions to align with the most appropriate role model. Through conceptual analysis, literature review, and the construction of a normative framework, the research uncovers implicit normative assumptions underlying existing methods, clarifies the hazards arising from role confusion, and establishes clear normative criteria for selecting and designing annotation protocols. This contribution advances a new paradigm of dimension-specific annotation mechanism design in RLHF.

This work proposes a low-barrier, real-time analysis system based on large language models to address the challenge user experience practitioners face in efficiently leveraging open-ended textual feedback. Designed within organizational and technical constraints, the system employs model fine-tuning and engineering optimizations to automatically classify user feedback and extract thematic insights, enabling non-technical stakeholders to independently access real-time analytical results. Evaluation of the prototype demonstrates that this approach substantially lowers the technical threshold for data analysis, enhances the accessibility and efficiency of deriving actionable insights from user feedback, and advances the practice of data democratization within organizations.

This study addresses the limitations of large language models (LLMs) in annotating complex social science constructs—such as climate mitigation pessimism—where autonomous labeling often yields suboptimal quality. To overcome this, the authors propose AnnotateThis, a human-centered interactive annotation system that introduces an innovative “LLM grounding” paradigm, deeply integrating expert knowledge into the LLM annotation pipeline. The system enables iterative co-evolution of conceptual definitions and model refinement through human–AI collaboration, interactive visualizations, and dynamic prompt optimization, functioning effectively both with and without ground-truth labels. Empirical evaluation demonstrates that, in labeled settings, AnnotateThis achieves a 0.15 improvement in F-Measure and a 0.23 gain in accuracy, significantly outperforming existing fully automated approaches.

Current AI evaluation practices relying on human judgment are susceptible to anchoring effects and lack scalability, limiting their ability to provide reliable quality signals. This work proposes a human–AI collaborative evaluation framework in which humans focus on identifying salient information units—referred to as “nuggets”—and making value judgments, while large language models (LLMs) efficiently match model outputs against these nuggets. The approach integrates human oversight and automated scoring through an interactive annotation tool, a three-stage workflow, and an exportable nugget repository. By structuring human input around discrete, reusable semantic units, the framework substantially improves evaluation consistency, scalability, and accountability, thereby enhancing the reliability of LLM-as-a-Judge paradigms.