Tutorial Debriefing: Applied Statistical Causal Inference in Requirements Engineering

2025-11-05

AI Summary

Randomized controlled trials (RCTs) are often infeasible in software engineering, hindering rigorous causal assessment of tools, processes, or guidelines on development outcomes (e.g., efficiency, quality, user experience). Method: We propose a statistical causal inference methodology grounded in observational data, integrating the potential outcomes framework, propensity score matching, and difference-in-differences to systematically address confounding bias and selection bias. Contribution/Results: This work pioneers the systematic application of formal causal inference paradigms to requirements engineering and software practice research, tailoring analytical workflows and evaluation criteria to the characteristics of software engineering data. Empirical validation demonstrates that our approach substantially improves internal validity and reproducibility of causal conclusions in non-experimental settings. By enabling robust, evidence-based causal claims from real-world development data, it strengthens the empirical foundation for translating research findings into industrial practice.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

As any scientific discipline, the software engineering (SE) research community strives to contribute to the betterment of the target population of our research: software producers and consumers. We will only achieve this betterment if we manage to transfer the knowledge acquired during research into practice. This transferal of knowledge may come in the form of tools, processes, and guidelines for software developers. However, the value of these contributions hinges on the assumption that applying them causes an improvement of the development process, user experience, or other performance metrics. Such a promise requires evidence of causal relationships between an exposure or intervention (i.e., the contributed tool, process or guideline) and an outcome (i.e., performance metrics). A straight-forward approach to obtaining this evidence is via controlled experiments in which a sample of a population is randomly divided into a group exposed to the new tool, process, or guideline, and a control group. However, such randomized control trials may not be legally, ethically, or logistically feasible. In these cases, we need a reliable process for statistical causal inference (SCI) from observational data.

Problem

• Establishing causal links between interventions and outcomes in software engineering
• Providing reliable statistical causal inference methods for observational data
• Addressing limitations of randomized controlled trials in practical research settings

Innovation

• Statistical causal inference from observational data
• Reliable process for causal relationship evidence
• Applied in software engineering research transfer