LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations

2025-11-18

AI Summary

This study addresses methodological challenges arising from the integration of large language models (LLMs) into qualitative theme analysis in software engineering (SE), specifically concerning rigor, transparency, and researcher subjectivity. Employing reflexive workshops that combine structured discussion with a color-coded canvas tool, we systematically examine LLM applications across open coding, theme generation, and thematic review. We find that LLMs cannot substitute for researchers’ interpretive judgment; rather, their value lies in enhancing analytical efficiency and scalability. A key contribution is the introduction of “prompt literacy”—a conceptual framework emphasizing sustained human oversight and procedural transparency. Furthermore, this work provides the first systematic articulation of opportunities, limitations, and responsible-use guidelines for LLM-assisted thematic analysis in SE. Collectively, it establishes a methodological framework and practical guidance for ethically integrating AI into qualitative SE research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

[Context] Large Language Models (LLMs) are increasingly used to assist qualitative research in Software Engineering (SE), yet the methodological implications of this usage remain underexplored. Their integration into interpretive processes such as thematic analysis raises fundamental questions about rigor, transparency, and researcher agency. [Objective] This study investigates how experienced SE researchers conceptualize the opportunities, risks, and methodological implications of integrating LLMs into thematic analysis. [Method] A reflective workshop with 25 ISERN researchers guided participants through structured discussions of LLM-assisted open coding, theme generation, and theme reviewing, using color-coded canvases to document perceived opportunities, limitations, and recommendations. [Results] Participants recognized potential efficiency and scalability gains, but highlighted risks related to bias, contextual loss, reproducibility, and the rapid evolution of LLMs. They also emphasized the need for prompting literacy and continuous human oversight. [Conclusion] Findings portray LLMs as tools that can support, but not substitute, interpretive analysis. The study contributes to ongoing community reflections on how LLMs can responsibly enhance qualitative research in SE.

Problem

• Investigating methodological implications of LLMs in qualitative software engineering research
• Assessing risks of bias and contextual loss in LLM-assisted thematic analysis
• Establishing recommendations for responsible LLM integration while preserving researcher agency

Innovation

• Using structured workshops for LLM-assisted qualitative analysis
• Applying color-coded canvases to document researcher perspectives
• Developing prompting literacy and human oversight methods