GenAI Is No Silver Bullet for Qualitative Research in Software Engineering

2026-03-09

AI Summary

This study systematically investigates the applicability boundaries and risks of misuse of generative artificial intelligence (GenAI) in qualitative software engineering research, emphasizing that GenAI is not a universal solution. By examining the multidimensional nature of qualitative inquiry and accounting for diverse research strategies and data characteristics, the work integrates large language model applications, qualitative data analysis, and empirical evidence to reconstruct quality assessment criteria. It elucidates the critical mechanisms governing the alignment between technological capabilities and methodological requirements. The research delineates specific scenarios where GenAI offers advantages as well as contexts prone to pitfalls, proposes principled guidelines for appropriate adoption, and offers practical guidance for researchers while charting directions for future investigation.

Technology Category

• Knowledge Representation and Reasoning: Qualitative Reasoning
• Philosophy and Ethics of AI: Artificial General Intelligence
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Qualitative research gives rich insights into the quintessentially human aspects of software engineering as a socio-technical system. Qualitative research spans diverse strategies and methods, from interpretivist, in situ observational field studies, to deductive coding of data from mining studies. Advances in large language models and generative AI (GenAI) have prompted claims that artificial intelligence could automate qualitative analysis. Such claims are overgeneralizing from narrow successes. GenAI support must be carefully adapted to the data of interest, but also to the characteristics of a particular research strategy. In this Frontiers of SE paper, we discuss the emerging use of GenAI in relation to the broad spectrum of qualitative research in software engineering. We outline the dimensions of qualitative work in software engineering, review emerging empirical evidence for GenAI assistance, examine the pros and cons of GenAI-mediated qualitative research practices, and revisit qualitative research quality factors, in light of GenAI. Our goal is to inform researchers about the promises and pitfalls of GenAI-assisted qualitative research. We conclude with future plans to advance understanding of its use in software engineering.

Problem

• Generative AI
• Qualitative Research
• Software Engineering
• Research Methodology
• AI Limitations

Innovation

• Generative AI
• Qualitative Research
• Software Engineering
• Research Methodology
• LLM