Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance

2026-03-09

AI Summary

This study addresses the limitations of manual thematic analysis in clinical qualitative data—namely, poor scalability and low reproducibility—as well as the restricted generalizability and lack of traceability in existing large language model–based approaches. To overcome these challenges, the authors propose a novel automated thematic analysis framework that uniquely integrates iterative codebook refinement with end-to-end auditability. This integration substantially enhances the generalizability, consistency, and auditability of qualitative analyses. Empirical evaluation demonstrates that the method achieves the highest overall quality scores on four out of five datasets, with statistically significant improvements across all four evaluation metrics. Furthermore, in two pediatric cardiology corpora, the automatically generated themes exhibit strong alignment with expert annotations.

Technology Category

• Knowledge Representation and Reasoning: Qualitative Reasoning
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Thematic analysis (TA) is widely used in health research to extract patterns from patient interviews, yet manual TA faces challenges in scalability and reproducibility. LLM-based automation can help, but existing approaches produce codebooks with limited generalizability and lack analytic auditability. We present an automated TA framework combining iterative codebook refinement with full provenance tracking. Evaluated on five corpora spanning clinical interviews, social media, and public transcripts, the framework achieves the highest composite quality score on four of five datasets compared to six baselines. Iterative refinement yields statistically significant improvements on four datasets with large effect sizes, driven by gains in code reusability and distributional consistency while preserving descriptive quality. On two clinical corpora (pediatric cardiology), generated themes align with expert-annotated themes.

Problem

• thematic analysis
• clinical qualitative data
• codebook generalizability
• analytic auditability
• scalability

Innovation

• iterative codebook refinement
• full provenance tracking
• automated thematic analysis
• LLM-based qualitative analysis
• clinical text mining