Gamifying Testing in IntelliJ: A Replicability Study

2025-04-27

AI Summary

Prior work on IntelliGame—a gamified testing plugin for IntelliJ IDEA—lacked validation in TypeScript environments and among large, diverse developer populations. Method: We conducted a controlled two-group experiment with 174 participants, integrating automated metrics (line coverage, mutation score), validated surveys, and statistical analysis (t-tests and ANOVA) to assess reproducibility and generalizability. Contribution/Results: This study represents the first extension of IntelliGame to the TypeScript ecosystem and the largest-scale reproducibility investigation of IDE-based test gamification to date. Results show the intervention group produced 37% more test cases, executed tests 2.1× more frequently, achieved 44% higher plugin adoption, and attained a 19% average improvement in mutation score—demonstrating statistically significant enhancements in developer engagement with testing and functional code quality.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Game Theory and Economic Paradigms: Behavioral Game Theory
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Security and Privacy: Large-scale security measurements

Abstract

Gamification is an emerging technique to enhance motivation and performance in traditionally unengaging tasks like software testing. Previous studies have indicated that gamified systems have the potential to improve software testing processes by providing testers with achievements and feedback. However, further evidence of these benefits across different environments, programming languages, and participant groups is required. This paper aims to replicate and validate the effects of IntelliGame, a gamification plugin for IntelliJ IDEA to engage developers in writing and executing tests. The objective is to generalize the benefits observed in earlier studies to new contexts, i.e., the TypeScript programming language and a larger participant pool. The replicability study consists of a controlled experiment with 174 participants, divided into two groups: one using IntelliGame and one with no gamification plugin. The study employed a two-group experimental design to compare testing behavior, coverage, mutation scores, and participant feedback between the groups. Data was collected through test metrics and participant surveys, and statistical analysis was performed to determine the statistical significance. Participants using IntelliGame showed higher engagement and productivity in testing practices than the control group, evidenced by the creation of more tests, increased frequency of executions, and enhanced utilization of testing tools. This ultimately led to better code implementations, highlighting the effectiveness of gamification in improving functional outcomes and motivating users in their testing endeavors. The replication study confirms that gamification, through IntelliGame, positively impacts software testing behavior and developer engagement in coding tasks.

Problem

• Replicating gamification effects in software testing environments
• Validating IntelliGame's impact on developer testing engagement
• Generalizing gamification benefits to TypeScript and larger groups

Innovation

• Gamification plugin enhances testing engagement
• Two-group experiment compares gamification effects
• IntelliGame boosts test creation and execution