An Empirical Study of Gemini 3 for Detecting Natural Language Test Smells in Manual Test Cases

2026-06-11

AI Summary

This work addresses the prevalence of test smells—such as ambiguity, redundancy, and missing assertions—in manually written test cases, which undermine their reliability and maintainability. Existing approaches rely on handcrafted rules with limited generalizability. To overcome these limitations, this study presents the first application of the GEMINI-3-PRO-PREVIEW large language model to detect test smells at the entire test-case level. Through prompt engineering and end-to-end analysis, the method holistically models inter-step dependencies to identify seven categories of test smells. Evaluated on 100 Ubuntu test cases, the approach significantly outperforms prior small-model methods, detecting nearly one smell per test step on average and generating actionable improvement suggestions. This advances beyond previous techniques that analyze steps in isolation, offering a more comprehensive and context-aware assessment of test quality.

Abstract

Manual testing, in which testers follow natural language instructions to validate system behavior, remains essential for uncovering issues that are difficult to capture with automation. However, manual test cases often contain test smells, quality issues such as ambiguity, redundancy, or missing checks that reduce reliability, maintainability, and reproducibility. Existing detection approaches largely depend on manually engineered rules and thus struggle to generalize and scale across heterogeneous test suites. In our previous work, we assessed the feasibility of using Small Language Models (SLMs) for test smell detection by evaluating GEMMA-3-4B, LLAMA-3.2-3B, and PHI-4-14B on test steps from 143 real-world Ubuntu test cases, covering seven smell types. PHI-4-14B achieved the best performance. In this article, we investigate whether a contemporary Large Language Model (GEMINI-3-PRO-PREVIEW) available at the time of the study can identify test smells in natural language manual test cases using a prompt-based, whole-test-case analysis strategy. Unlike approaches that analyze individual test steps in isolation, our approach evaluates complete test cases, enabling the model to consider relationships and dependencies among test steps. We evaluate the approach on 100 Ubuntu test cases covering seven test smell types and compare its performance against previously evaluated SLMs, including GEMMA-3-4B, LLAMA-3.2-3B, and PHI-4-14B. Our results show that GEMINI-3-PRO-PREVIEW outperforms the SLMs, while producing actionable explanations that can help practitioners revise manual test cases for greater clarity and consistency. We also find that test smells are pervasive in practice, with nearly one detected test smell per step on average, highlighting the need for scalable and automated quality support for manual testing artifacts.

Problem

• test smells
• manual testing
• natural language test cases
• software quality
• test case analysis

Innovation

• Large Language Model
• Test Smell Detection
• Prompt-based Analysis
• Manual Test Cases
• Whole-test-case Evaluation