SFT-TA: Supervised Fine-Tuned Agents in Multi-Agent LLMs for Automated Inductive Thematic Analysis

2025-09-21

AI Summary

Manual thematic analysis of clinical interview transcripts is time-consuming and inefficient, while existing large language model (LLM)-based automated approaches exhibit insufficient agreement with human annotations. To address these limitations, this paper proposes a supervised fine-tuning (SFT)-driven multi-agent LLM framework specifically designed for inductive thematic analysis of clinical interview texts. The framework explicitly assigns distinct roles—such as coder, integrator, and verifier—to specialized SFT agents, embedding them within a collaborative workflow that enhances semantic understanding and enables iterative validation. Experimental results demonstrate that our method significantly outperforms GPT-4o and mainstream single-agent or unsupervised baselines in thematic consistency metrics, including Cohen’s κ and F1-score. These findings validate the efficacy of SFT-powered multi-agent architectures in improving both the reliability and interpretability of automated thematic analysis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Modeling other Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Thematic Analysis (TA) is a widely used qualitative method that provides a structured yet flexible framework for identifying and reporting patterns in clinical interview transcripts. However, manual thematic analysis is time-consuming and limits scalability. Recent advances in LLMs offer a pathway to automate thematic analysis, but alignment with human results remains limited. To address these limitations, we propose SFT-TA, an automated thematic analysis framework that embeds supervised fine-tuned (SFT) agents within a multi-agent system. Our framework outperforms existing frameworks and the gpt-4o baseline in alignment with human reference themes. We observed that SFT agents alone may underperform, but achieve better results than the baseline when embedded within a multi-agent system. Our results highlight that embedding SFT agents in specific roles within a multi-agent system is a promising pathway to improve alignment with desired outputs for thematic analysis.

Problem

• Manual thematic analysis of clinical interviews is time-consuming and limits scalability
• Existing LLM approaches for automated thematic analysis lack alignment with human results
• Individual supervised fine-tuned agents may underperform without proper system integration

Innovation

• Uses supervised fine-tuned agents in multi-agent system
• Embeds SFT agents in specific roles for thematic analysis
• Combines SFT with multi-agent architecture for better alignment