From First Patch to Long-Term Contributor: Evaluating Onboarding Recommendations for OSS Newcomers

2024-07-04

arXiv.org

AI Summary

This study addresses the lack of generalizability in onboarding recommendations for newcomers to open-source projects and the conflation of their needs with those of long-term contributors. Through a systematic literature review (SLR), we distilled 15 actionable onboarding recommendations and conducted a large-scale, cross-platform (Gerrit/GitHub) empirical analysis across 1,160 real-world projects. We innovatively identified four types of recommendation effects: universally positive, context-dependent, universally negative, and newcomer-exclusive. Notably, we introduce the “timely abandonment of newcomer-specific strategies” paradigm, emphasizing stage-adaptivity in contributor guidance. Results show that four recommendations significantly increase first-patch acceptance rates, while three are beneficial only during the newcomer phase—continued application reduces long-term retention. This work provides both theoretical grounding and a practical framework for precision, lifecycle-aware onboarding of open-source contributors.

Technology Category

• Data Mining & Knowledge Management: Recommender Systems
• Application Domains: Humanities & Computational Social Science
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Consent frameworks and practices on the web
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Attracting and retaining a steady stream of new contributors is crucial to ensuring the long-term survival of open-source software (OSS) projects. However, there are two key research gaps regarding recommendations for onboarding new contributors to OSS projects. First, most of the existing recommendations are based on a limited number of projects, which raises concerns about their generalizability. If a recommendation yields conflicting results in a different context, it could hinder a newcomer's onboarding process rather than help them. Second, it's unclear whether these recommendations also apply to experienced contributors. If certain recommendations are specific to newcomers, continuing to follow them after their initial contributions are accepted could hinder their chances of becoming long-term contributors. To address these gaps, we conducted a two-stage mixed-method study. In the first stage, we conducted a Systematic Literature Review (SLR) and identified 15 task-related actionable recommendations that newcomers to OSS projects can follow to improve their odds of successful onboarding. In the second stage, we conduct a large-scale empirical study of five Gerrit-based projects and 1,155 OSS projects from GitHub to assess whether those recommendations assist newcomers' successful onboarding. Our results suggest that four recommendations positively correlate with newcomers' first patch acceptance in most contexts. Four recommendations are context-dependent, and four indicate significant negative associations for most projects. Our results also found three newcomer-specific recommendations, which OSS joiners should abandon at non-newcomer status to increase their odds of becoming long-term contributors.

Problem

• Evaluate onboarding recommendations for OSS newcomers.
• Assess generalizability of recommendations across diverse OSS projects.
• Identify recommendations hindering long-term contributor development.

Innovation

• Conducted Systematic Literature Review for recommendations
• Empirical study on Gerrit and GitHub projects
• Identified context-dependent and newcomer-specific recommendations