A Multi-Agent Large Language Model Framework for Automated Qualitative Analysis

2025-12-17

AI Summary

Chronic disease patient experience research relies heavily on qualitative thematic analysis, yet conventional approaches are labor-intensive, highly subjective, and difficult to scale. To address these limitations, we propose CoTI—the first collaborative multi-agent large language model framework designed specifically for qualitative analysis. CoTI comprises three specialized agents—Instructor, Thematizer, and CodebookGenerator—that jointly perform automated theme extraction, dynamic codebook generation, and alignment with expert judgment. Leveraging role-based prompt engineering, semantic similarity evaluation, and integrated human-in-the-loop interfaces, CoTI significantly enhances analytical efficiency and inter-rater consistency. Empirical evaluation on heart failure patient interviews demonstrates that CoTI’s thematic outputs achieve significantly higher agreement with senior researchers than those produced by junior researchers or baseline NLP models. The framework has been implemented as a deployable, production-ready analytical tool.

Technology Category

• Knowledge Representation and Reasoning: Qualitative Reasoning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Multiagent Systems: Teamwork

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Understanding patients experiences is essential for advancing patient centered care, especially in chronic diseases that require ongoing communication. However, qualitative thematic analysis, the primary approach for exploring these experiences, remains labor intensive, subjective, and difficult to scale. In this study, we developed a multi agent large language model framework that automates qualitative thematic analysis through three agents (Instructor, Thematizer, CodebookGenerator), named Collaborative Theme Identification Agent (CoTI). We applied CoTI to 12 heart failure patient interviews to analyze their perceptions of medication intensity. CoTI identified key phrases, themes, and codebook that were more similar to those of the senior investigator than both junior investigators and baseline NLP models. We also implemented CoTI into a user-facing application to enable AI human interaction in qualitative analysis. However, collaboration between CoTI and junior investigators provided only marginal gains, suggesting they may overrely on CoTI and limit their independent critical thinking.

Problem

• Automates qualitative thematic analysis of patient experiences using multi-agent LLM framework.
• Addresses labor-intensive, subjective, and scalability issues in analyzing chronic disease patient interviews.
• Evaluates AI-human collaboration impact on critical thinking in qualitative research.

Innovation

• Multi-agent LLM framework automates qualitative analysis
• Three specialized agents identify themes and generate codebooks
• Application enables AI-human interaction in thematic analysis