frameworks

This work addresses the lack of standardized data for migrating Python projects from unittest to pytest, a gap that has hindered the development and evaluation of automated migration tools. To bridge this gap, the authors present TestMigrationsInPy, the first structured dataset comprising 923 real-world migration instances systematically collected and annotated from historical commits in open-source projects. The dataset provides fine-grained categorization of migration patterns—such as assertions and fixtures—enabling precise analysis and comparison of transformation strategies. Publicly released, TestMigrationsInPy establishes the first benchmark for automated test migration research, facilitating quantitative evaluation of diverse migration approaches and supporting future advancements in tooling for Python testing frameworks.

This work proposes an automated approach leveraging large language models (LLMs) to migrate unittest-based test suites in Python projects to Pytest, aiming to reduce manual effort and accelerate test modernization. We introduce the first real-world dataset for unittest-to-Pytest migration and systematically evaluate the performance of GPT-4o and Claude Sonnet 4 under zero-shot, one-shot, and chain-of-thought prompting strategies. Experimental results show that 48.5% of LLM-generated migrations pass the original test suite: GPT-4o tends to aggressively refactor tests into a functional style, whereas Claude Sonnet 4 conservatively preserves the original class-based structure. These findings demonstrate the potential of LLMs as assistants in test migration tasks while underscoring the necessity of careful validation of their outputs.

This study addresses the lack of empirical evidence on the real-world impact of UI testing frameworks in CI/CD pipelines. Using GitHub API data collection, YAML configuration parsing, CI log metric extraction, and controlled time-series analysis across open-source repositories, we systematically quantify the integration patterns and effects of Selenium, Playwright, and Cypress within GitHub Actions workflows. Results show that UI testing significantly improves test pass-rate stability but increases mean build duration by 12% initially. Highly active repositories prefer Playwright—its built-in retry mechanism reduces flaky-test-induced pipeline interruptions by 35%. This work fills a critical gap in understanding UI testing’s practical implications in production CI/CD environments, providing data-driven insights for quality assurance strategy design and framework selection.

This study addresses the imbalance in the test pyramid—characterized by an overreliance on coarse-grained integration and system tests, which leads to difficulties in fault localization and slow execution—by proposing, for the first time, a method to automatically generate unit tests from existing integration tests. The approach combines static and dynamic analysis to automatically isolate component dependencies and enhance coverage at the unit level. Implemented as a Node.js tool and evaluated on twelve open-source JavaScript projects, the technique produces high-quality unit tests that significantly improve test suite structure, thereby increasing both testing efficiency and maintainability.

Deep learning framework testing suffers from the oracle problem, insufficient metamorphic relations (MRs) that overlook model structural complexity and multi-interface compositions, and poor generalizability. To address these challenges, we propose ModelMeta, a model-level mutation testing method. ModelMeta introduces four novel MRs derived from structural characteristics of deep learning models, explicitly covering multi-interface composition scenarios. It employs a QR-DQN–guided strategy to enhance seed models and generate structurally equivalent mutated inputs. Fine-grained divergence detection is performed across multiple runtime dimensions—including training loss/gradients, memory/GPU utilization, and execution time. Evaluated on mainstream frameworks, ModelMeta significantly improves test input diversity and defect detection rates, uncovering several previously unknown cross-interface inconsistencies. Our results demonstrate both the effectiveness and practical applicability of ModelMeta in enhancing deep learning framework reliability.

This work addresses the lack of systematic validation for unit tests generated by foundation models, which hinders reliable assessment of their correctness, utility, and maintainability. To bridge this gap, we introduce TestMap—an open-source infrastructure tailored for C#/.NET projects—that establishes an evidence-centered framework for automated test generation across its entire lifecycle, encompassing mapping, execution, repair, evaluation, and experiment tracking. By integrating repository analysis, source-test mapping, coverage and mutation testing, static analysis, test smell detection, and model-guided generation, TestMap enables observable, reproducible, and comparable experimentation across diverse models, prompts, and strategies. The framework further uncovers limitations of current models, missing contextual information, repair overhead, and latent defects in the system under test, thereby providing an empirical foundation for trustworthy test generation.

This work addresses the oracle problem in REST API testing—stemming from the absence of explicit expected outputs—by proposing a novel multi-agent workflow grounded in large language models (LLMs). The approach uniquely integrates multi-agent LLMs with metamorphic testing to automatically parse OpenAPI specifications and generate executable test cases in the Given-When-Then format, eliminating the need for manually defined oracles. Evaluated on two public web applications, the method demonstrates its effectiveness by uncovering API behaviors missed by existing scenario-based testing techniques and substantially enhancing test coverage. This study thus introduces a new paradigm for automated API testing that leverages the reasoning and coordination capabilities of LLM-based agents to overcome longstanding challenges in test oracle specification.

This study investigates how the syntactic arrangement of test code—whether co-located with or separated from implementation code—affects the quality of code generated by large language models. Through a large-scale empirical analysis, it compares inline tests (e.g., Python doctests) against separated tests (e.g., Rust #[test] attributes), evaluating model outputs using the SEGA framework across three dimensions: determinism, retention, and correctness. The work reveals, for the first time, that co-locating tests consistently and significantly enhances code quality across twelve prominent models, including six Transformer variants and one gated linear RNN. Results show inline tests achieve near-perfect retention rates (≈100%) and high correctness (92–100%), whereas separated tests not only expose performance disparities among models but also decouple correctness from retention, offering critical insights for future model design.

This work addresses the common failure of large language models (LLMs) to generate necessary test fixtures when automatically producing unit tests, which often leads to invalid or ineffective test cases. To overcome this limitation, the authors propose Fixturize, a novel framework that introduces fixture-awareness for the first time through a diagnostic identification and iterative feedback mechanism to actively synthesize missing test fixtures in coordination with LLMs and search-based testing tools. The study contributes FixtureEval, the first benchmark annotated with fixture dependencies, thereby filling a critical gap in automated testing pipelines. Evaluated on Python and Java, Fixturize achieves fixture dependency identification accuracy of 88.38%–97.00%, improves SuitePS by 18.03%–42.86%, and increases line and branch coverage by up to 31.54% and 119.66%, respectively.

This study challenges the common assumption that high code coverage implies high fault localization accuracy in spectrum-based fault localization (SBFL). We systematically evaluate the effectiveness of automatically generated test cases—produced by tools such as EvoSuite and Randoop—against manually written tests, measuring SBFL score (our primary quality metric), mutation kill rate, and branch coverage across 42 real-world defects from Defects4J. Contrary to expectations, although automatically generated tests achieve 18% higher average branch coverage, they yield 23% lower SBFL scores. Moreover, in deeply nested code regions, their fault localization precision degrades by up to 41%. These findings reveal a fundamental inconsistency between coverage metrics and actual fault localization capability. The work provides the first empirical evidence centered on SBFL score as the key evaluation criterion, offering critical support for hybrid testing strategies and advocating a paradigm shift in test quality assessment—from coverage-oriented to localization-oriented evaluation.