Automatic Bias Detection in Source Code Review

2025-04-25

AI Summary

Implicit human biases in code review often lead to unfair decisions, yet existing methods struggle to detect such biases in real time. Method: This paper proposes a novel eye-tracking–based bias detection paradigm for code review—the first to introduce the “attention spotlight model” into this domain. By capturing reviewers’ gaze sequences, we jointly model attention dynamics using Markov models, recurrent neural networks (RNNs), and conditional random fields (CRFs), enabling online bias identification without relying on post-hoc analysis of review outcomes or textual content—and crucially, without requiring manual annotation of reviewer intent. Contribution/Results: Experimental results demonstrate that the extracted gaze sequence patterns are significantly correlated with biased rejections. Our approach effectively reduces spurious rejection rates, thereby enhancing fairness in code review and fostering greater inclusivity within development teams.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Bias is an inherent threat to human decision-making, including in decisions made during software development. Extensive research has demonstrated the presence of biases at various stages of the software development life-cycle. Notably, code reviews are highly susceptible to prejudice-induced biases, and individuals are often unaware of these biases as they occur. Developing methods to automatically detect these biases is crucial for addressing the associated challenges. Recent advancements in visual data analytics have shown promising results in detecting potential biases by analyzing user interaction patterns. In this project, we propose a controlled experiment to extend this approach to detect potentially biased outcomes in code reviews by observing how reviewers interact with the code. We employ the"spotlight model of attention", a cognitive framework where a reviewer's gaze is tracked to determine their focus areas on the review screen. This focus, identified through gaze tracking, serves as an indicator of the reviewer's areas of interest or concern. We plan to analyze the sequence of gaze focus using advanced sequence modeling techniques, including Markov Models, Recurrent Neural Networks (RNNs), and Conditional Random Fields (CRF). These techniques will help us identify patterns that may suggest biased interactions. We anticipate that the ability to automatically detect potentially biased interactions in code reviews will significantly reduce unnecessary push-backs, enhance operational efficiency, and foster greater diversity and inclusion in software development. This approach not only helps in identifying biases but also in creating a more equitable development environment by mitigating these biases effectively

Problem

• Detect biases in code reviews automatically
• Analyze reviewer gaze patterns for bias indicators
• Improve diversity and inclusion in software development

Innovation

• Uses gaze tracking to detect reviewer focus areas
• Applies sequence modeling techniques like RNNs and CRFs
• Leverages spotlight model for bias pattern identification