Revisiting the Replication Study Design Used in Computing Education Research

2026-03-28

AI Summary

This study addresses the persistently low prevalence and publication barriers of replication studies in Computing Education Research (CER), which hinder the reliability and cumulative progress of knowledge in the field. For the first time, it conducts a longitudinal analysis of replication efforts in CER from 2019 to 2025, combining a systematic literature review with a reproducibility-oriented survey across five major conferences and journals to examine trends and researchers’ attitudes and practices. Findings indicate a modest increase in replication studies—still overall limited—with most appearing in journals and predominantly consisting of self-replications. Although practitioners frequently engage in replication informally, significant obstacles remain to formal publication. This work provides the first temporal evidence of the replication landscape in CER and highlights critical opportunities for institutional and cultural reform.

Abstract

Replication studies play an important role in Computing Education Research (CER) by supporting the development of consistent and reliable scientific knowledge. However, prior research indicates that the CER community tends to prioritise novel contributions over replication. A 2019 SLR identified only 54 replication studies among 2,269 papers published between 2009 and 2018 across five major CER venues. This study aims to examine how the landscape of replication research in CER has evolved since 2019. Specifically, we investigate whether the prevalence of replication studies has increased and explore current perceptions and experiences of CER researchers regarding replication. We replicated two prior studies. First, we conducted an updated SLR to identify replication studies published between 2019 and 2025 in the same five CER venues. Second, we replicated a survey of Computing Education researchers to better understand their perceptions, experiences, and challenges related to conducting and publishing replication studies. Our SLR identified 63 replication studies among 2,516 published papers. While the proportion of replication studies has increased slightly, overall growth remains limited. We observed a shift toward more published replication studies in journals and an increase in authors replicating their own prior work. Survey results indicate that although many researchers engage in replication within their teaching and research practice, they encounter significant challenges when attempting to publish replication studies. Despite increased discourse around open science and research rigour, the adoption of replication studies in CER has not substantially grown. Our findings offer opportunities for future research to promote replication and to explore how the CER community can encourage researchers to publish replication studies.

Problem

• replication studies
• Computing Education Research
• research reproducibility
• publication bias
• open science

Innovation

• replication study
• computing education research
• systematic literature review
• research reproducibility
• meta-science