Test Case Selection for Deep Neural Networks: A Replication Study on LLMs for Code

2026-06-25

AI Summary

Existing test case selection (TCS) methods have demonstrated strong performance in vision models, yet their applicability to code-focused large language models (LLMs) remains unclear. This study presents the first systematic reproduction and evaluation of 13 TCS strategies—including 12 feature-aware approaches and random sampling—across three software engineering tasks: clone detection, vulnerability identification, and technical debt prediction. Leveraging 17 fine-tuned code LLM instances and seven predictive features, our large-scale empirical analysis reveals that uncertainty-based features are more effective for early failure detection, while representation-based features better support accuracy estimation. However, TCS effectiveness is highly dependent on both the specific task and model architecture, with only limited insights from vision-domain studies generalizing to code LLMs. These findings underscore the context-sensitive nature of TCS in the code LLM setting.

Abstract

Recently, test case selection (TCS) techniques have been explored to support the operational evaluation of deep neural networks (DNNs) under limited testing budgets, where labeling cost is a primary concern and uncovering model failures early is a key objective. Although prior studies report promising results, existing empirical evaluations focus almost exclusively on vision-based DNNs and datasets, leaving it unclear whether prior findings generalize to LLM code models. This paper presents a large-scale replication study of TCS techniques in the context of LLM code models. We re-examine established TCS strategies originally proposed for DNNs and complement them with statistical sampling strategies not previously evaluated for TCS. We assess their effectiveness on three code-related classification tasks: clone detection, vulnerability detection, and technical debt prediction. The study spans 17 task-specific fine-tuned model instances, 7 predictive features, and 13 selection strategies, including 12 feature-aware strategies and simple random sampling (SRS) as a feature-agnostic baseline. We evaluate performance along two dimensions: accuracy estimation and early failure discovery. The results indicate that only a subset of findings reported for vision-based DNNs generalize when TCS is applied to LLMs for code. In particular, uncertainty-based features are effective for early failure discovery, while representation-based features are more robust for accuracy estimation. At the same time, performance varies substantially across tasks and models, indicating that TCS effectiveness is context-dependent. Overall, this study provides empirical evidence on the replicability of TCS techniques beyond vision-based deep learning and offers insights into their use for the operational evaluation of LLMs for code.

Problem

• test case selection
• large language models
• code models
• replication study
• deep neural networks

Innovation

• test case selection
• large language models
• code models
• replication study
• operational evaluation