A Computational Method for Measuring "Open Codes" in Qualitative Analysis
🤖 AI Summary
Qualitative evaluation of inductive coding faces significant challenges: conventional metrics are ill-suited for exploratory processes; manual assessment is labor-intensive; and expert-annotated “ground truth” introduces methodological limitations. This paper introduces the first quantifiable evaluation framework for open coding in grounded theory and thematic analysis. Moving beyond the conventional human–AI alignment paradigm, it innovatively integrates stability assessment (Cohen’s kappa, Jaccard similarity) with cross-human–machine comparison as a dual-validation mechanism. Crucially, the framework operates without presupposing ground-truth labels, enabling bias detection and coding quality measurement in human–AI collaborative workflows. Evaluated on two HCI datasets, the framework demonstrates high inter-coder agreement (κ > 0.75), strong output stability (Jaccard similarity > 0.89 across repeated runs), and yields a reusable, AI-augmented coding workflow.
Application Category
📝 Abstract
Qualitative analysis is critical to understanding human datasets in many social science disciplines. Open coding is an inductive qualitative process that identifies and interprets"open codes"from datasets. Yet, meeting methodological expectations (such as"as exhaustive as possible") can be challenging. While many machine learning (ML)/generative AI (GAI) studies have attempted to support open coding, few have systematically measured or evaluated GAI outcomes, increasing potential bias risks. Building on Grounded Theory and Thematic Analysis theories, we present a computational method to measure and identify potential biases from"open codes"systematically. Instead of operationalizing human expert results as the"ground truth,"our method is built upon a team-based approach between human and machine coders. We experiment with two HCI datasets to establish this method's reliability by 1) comparing it with human analysis, and 2) analyzing its output stability. We present evidence-based suggestions and example workflows for ML/GAI to support open coding.
Problem
Research questions and friction points this paper is trying to address.
Measuring inductive coding quality in qualitative analysis
Evaluating human and AI-generated codes with computational metrics
Assessing coding contributions through coverage, overlap, novelty, divergence
Innovation
Methods, ideas, or system contributions that make the work stand out.
LLM-enriched algorithm for codebook merging
Four novel metrics for coder contribution evaluation
Computational method for inductive coding measurement
🔎 Similar Papers
No similar papers found.
