From Developer Pairs to AI Copilots: A Comparative Study on Knowledge Transfer

2025-06-05

AI Summary

This study investigates knowledge transfer between AI programming assistants (GitHub Copilot) and human developers, benchmarking it against traditional human–human pair programming. Method: Through a controlled experiment, an extended knowledge transfer analytical framework, and a semi-automated evaluation pipeline—integrating qualitative coding and topic modeling—we systematically quantify and compare the frequency, topical coverage, and underlying mechanisms of knowledge transfer across both collaboration paradigms. Contribution/Results: We provide the first empirical evidence that human–AI collaboration achieves knowledge transfer frequency and topical overlap comparable to human–human pairing. However, it exhibits a dual cognitive effect: (1) reduced critical scrutiny of AI suggestions—indicating trust bias—and (2) AI’s capacity to proactively surface salient yet easily overlooked code details—demonstrating cognitive augmentation. These findings constitute the first systematic, empirically grounded characterization of knowledge transfer mechanisms in AI-augmented programming.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Knowledge transfer is fundamental to human collaboration and is therefore common in software engineering. Pair programming is a prominent instance. With the rise of AI coding assistants, developers now not only work with human partners but also, as some claim, with AI pair programmers. Although studies confirm knowledge transfer during human pair programming, its effectiveness with AI coding assistants remains uncertain. To analyze knowledge transfer in both human-human and human-AI settings, we conducted an empirical study where developer pairs solved a programming task without AI support, while a separate group of individual developers completed the same task using the AI coding assistant GitHub Copilot. We extended an existing knowledge transfer framework and employed a semi-automated evaluation pipeline to assess differences in knowledge transfer episodes across both settings. We found a similar frequency of successful knowledge transfer episodes and overlapping topical categories across both settings. Two of our key findings are that developers tend to accept GitHub Copilot's suggestions with less scrutiny than those from human pair programming partners, but also that GitHub Copilot can subtly remind developers of important code details they might otherwise overlook.

Problem

• Comparing knowledge transfer in human-human vs human-AI pair programming
• Assessing effectiveness of GitHub Copilot in code knowledge transfer
• Evaluating developer scrutiny of AI vs human programming suggestions

Innovation

• Comparative study on human-human and human-AI knowledge transfer
• Extended framework and semi-automated evaluation pipeline
• Analyzed GitHub Copilot suggestion acceptance patterns