documentation

Reproducibility remains a critical challenge in large language model (LLM)-driven software engineering (SE) research, undermining credibility and cumulative scientific progress. Method: We conducted a systematic literature review of 640 papers, integrating structured metadata extraction, manual annotation, and cross-platform analysis to diagnose reproducibility deficiencies across code, data, execution environments, and version control. Contribution/Results: We propose a novel taxonomy of seven reproducibility defect categories and introduce the Reproducibility Maturity Model (RMM), shifting evaluation from binary “reproducible/not reproducible” to a multi-dimensional, incremental framework. Our findings reveal that even top-tier conferences’ artifact evaluation badges exhibit low enforcement fidelity and poor long-term reproducibility; publication venue transparency practices vary substantially. This work provides both a theoretical framework and empirical evidence to enhance the rigor and trustworthiness of LLM-SE research.

High documentation maintenance costs and incomplete coverage due to overreliance on source code alone hinder software documentation quality. To address this, this paper proposes a large language model–based, multi-source collaborative documentation generation method. Our core innovation is a hierarchical prompting mechanism that jointly leverages heterogeneous development artifacts—including source code, commit logs, issue reports, and test cases—drawn from the DocMine dataset, enabling semi-automated generation of structured API-level and file-level documentation. This approach transcends the limitations of code-only methods, significantly enhancing contextual completeness in generated documentation. Evaluation on a manually annotated test set shows that our method achieves up to 43.24 BLEU-4 and 0.39 ROUGE-L scores for API- and file-level documentation, respectively, while attaining approximately 30% BLEU-4 for other documentation types. These results empirically validate the effectiveness and generalizability of our multi-source fusion strategy.

This study addresses the longstanding fragmentation in software artifact traceability research, characterized by incomplete linkages, ambiguous techniques, and disconnected application contexts. Through a systematic literature review, it constructs the first comprehensive traceability landscape encompassing 22 artifact types and 23 relationship kinds, and introduces a technology decision map, a standardized evaluation benchmark, and a role-oriented dynamic path alignment framework. The work uncovers critical challenges: a pervasive code-centric bias, a reproducibility crisis stemming from only 37% of studies releasing open-source artifacts, and a significant adoption gap with 95% of proposed tools never deployed in industry. In response, it offers targeted strategies to bridge these gaps, establishing a unified knowledge foundation for future research and practical implementation in traceability.

This work addresses the high cost, labor intensity, and poor reproducibility inherent in manual evaluation of software engineering artifacts. To this end, we propose Artisan, an intelligent agent powered by large language models that formalizes research reproduction as a standalone code generation task for the first time. Artisan incorporates a novel automated evaluation mechanism that deliberately withholds ground-truth results to guide the agent toward accurate reproductions without leakage. We also introduce Artisan-Bench, the first benchmark specifically designed for automated assessment in software engineering. Experimental results demonstrate that Artisan successfully generates 44 correct reproduction scripts out of 60 tasks, achieving 3.14× the performance of baseline methods at an average cost of 0.45 hours per task, and uncovers 20 previously unknown errors in published papers or their associated artifacts.

Current research software publishing lacks automated tools to ensure FAIR (Findable, Accessible, Interoperable, Reusable) compliance. To address this, we propose HERMES—a CI-based, extensible software publishing workflow that automatically generates software artifacts enriched with persistent identifiers (PIDs) and structured metadata. Our contributions are threefold: (1) the first configurable, reusable publishing pipeline designed for the full lifecycle of research software; (2) a modular plugin architecture enabling flexible integration of heterogeneous metadata sources; and (3) a Python-driven CLI toolchain with native GitHub Actions support. Evaluated across three cross-domain empirical case studies, HERMES significantly reduces manual publishing effort, improves metadata completeness by up to 40%, and increases FAIR compliance rates by over 65%. The workflow provides foundational infrastructure for sustainable, standards-compliant research software publication.

Design-oriented visualization research often struggles to meet conventional reproducibility standards due to its inherent subjectivity, contextual dependence, and iterative nature, thereby limiting its transparency and rigor. To address this challenge, this work proposes “traceability” as a viable alternative to traditional reproducibility. It presents the first systematic theoretical framework centered on three core components—recording, reporting, and reading—and introduces tRRRacer, a supporting tool implementing this framework. Through collaborative autoethnography, the authors reflect on practical applications of traceability in design-oriented research, demonstrating its feasibility and yielding actionable principles alongside theoretical insights. This approach offers a novel pathway to enhance the rigor and transparency of such studies without relying on strict reproducibility criteria.

While existing OpenAPI specifications are structurally compliant, they often fail to meet the semantic requirements of AI agents for effective task planning and API invocation, leading to systematic failures. This work proposes Hermes, a multi-agent large language model system that pioneers the application of multi-agent LLMs to endpoint-level semantic readiness assessment of large-scale, production-grade APIs. Hermes analyzes OpenAPI specifications, identifies RESTful and documentation-related code smells, and generates interpretable diagnostic reports. Evaluated on 16 real-world APIs encompassing approximately 600 endpoints, Hermes detected 2,450 such smells, with high precision confirmed by developer validation. The findings have prompted enterprises to incorporate semantic documentation quality into their API governance frameworks and to redefine API design standards for AI agent compatibility.

This work addresses the challenges of manual Web API integration testing, which is time-consuming, error-prone, and often misaligned with business requirements. The authors propose a novel approach that synergistically combines large language models (LLMs), retrieval-augmented generation (RAG), and prompt engineering to jointly parse natural language business requirements and OpenAPI specifications, thereby automatically generating executable test scripts that are both semantically meaningful and syntactically correct. Evaluated on ten real-world APIs, the method successfully produced valid tests for 89% of the business requirements within three attempts, uncovered multiple previously unknown integration defects, and substantially reduced the manual effort required for test development.

Addressing challenges in FAIR principle implementation—including fragmented data and code lifecycles, lack of executable environments, and high technical barriers—this study proposes a unified open-science platform. The platform uniquely integrates version control, containerized computational environments, and modular project scaffolding to support end-to-end reproducible research, from grant proposal to publication. It interoperates with mainstream scientific toolchains, supports deployment on both local workstations and institutional servers, and provides a lightweight graphical user interface. Empirical validation demonstrates successful re-execution of over a dozen interdisciplinary studies published more than ten years ago, confirming the platform’s robust long-term reproducibility, cross-platform compatibility, and seamless execution across diverse domains. By significantly lowering technical adoption barriers for researchers, the platform enables practical integration of FAIR principles and reproducibility practices into routine scientific workflows.

This work addresses the challenge of reproducibility in data visualization scripts, which often lack essential components such as source code, input data, execution environment, or output artifacts. To bridge this gap, we propose yProv4DV—a lightweight Python library that, for the first time, targets script-based visualization workflows by automatically capturing comprehensive provenance information—including source code, input data, runtime environment, and output results—through a single function call. Designed to be minimally invasive and ready-to-use, yProv4DV enables full reproducibility of visualization outputs without requiring any modification to existing scripts. This approach significantly reduces the development burden on researchers striving to ensure reproducibility and fills a critical void in automated provenance support within visualization pipelines.