Can Recombination Displace Dominant Scientific Ideas

2025-06-19

AI Summary

The accelerating growth of scientific literature has not been matched by a corresponding increase in scientific breakthroughs, and the underlying generative mechanisms remain unclear. Method: Leveraging a corpus of 49 million papers published between 1960 and 2024, this study systematically tests the dominant “cross-domain recombination drives disruptive innovation” hypothesis using atypical recombination measures, the disruption index (DI), longitudinal citation analysis, version-controlled paper comparisons, and in-depth interviews with scientists. Contribution/Results: Contrary to prevailing theory, highly disruptive work arises predominantly from *intra-domain paradigm shifts*—specifically, conceptual displacement within established frameworks—rather than cross-domain recombination. Conceptual displacement consistently precedes methodological displacement; atypical recombination exhibits a robust negative correlation with disruption; and high-DI papers concentrate intensely on core cited topics. This study provides the first empirical evidence that intra-domain displacement, not knowledge recombination across domains, is the primary engine of scientific disruption—challenging a foundational assumption of knowledge recombination theory.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Knowledge Representation and Reasoning: Action, Change, and Causality
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Web Mining and Content Analysis: Interdisciplinary science discovery with web data mining
• Social Networks and Social Media: Computational social science
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Scientific breakthroughs are widely attributed to the novel recombination of existing ideas. Yet despite explosive global growth in scientific labor and publications -- creating more opportunities to reconfigure knowledge -- the rate of breakthroughs has not kept pace. To investigate this disconnect, we analyze 49 million scholarly works from 1960 to 2024 using measures of atypical recombination and disruptive innovation. Contrary to recombination theories, we find a robust negative correlation between atypicality and disruption -- consistent across fields, time, team sizes, and even across versions of the same paper. Interviews with scientists about landmark breakthroughs suggest an alternative mechanism of innovation: displacement, in which nearly all breakthroughs supplanted dominant ideas within their fields. Bibliometric evidence confirms this pattern, showing that disruptive work often overlap in topic with its most-cited reference, indicating that breakthroughs emerge through within -- field displacement rather than cross-domain recombination. Notably, displacing methods takes longer than displacing theories -- revealing different temporal dynamics of epistemic change.

Problem

• Examines why scientific breakthroughs lag despite increased recombination opportunities
• Investigates negative link between atypical recombination and disruptive innovation
• Proposes displacement of dominant ideas as primary breakthrough mechanism

Innovation

• Analyzes atypical recombination and disruption
• Identifies displacement as key breakthrough mechanism
• Shows displacing methods takes longer than theories