Discovery under Hypothesis Redundancy: A Geometric Theory of Discovery Bottlenecks

2026-06-12

AI Summary

Scientific discovery often stagnates due to hypothesis redundancy, wherein even vast hypothesis spaces yield new proposals that provide little independent information. This work proposes the “search compression hypothesis,” asserting that non-local exploration guided by large language models (LLMs) is effective only when three geometric conditions are met: spectral compression, orthogonal escape, and residual signal alignment. We introduce a novel diagnostic framework that translates LLM-generated hypotheses into subspace geometry, validated through spectral analysis, signal implantation experiments, and both synthetic and real-world data (including A-share factor discovery and symbolic regression). Results demonstrate that hybrid search strategies enhance discovery efficiency exclusively along weakly represented directions containing target signals, with gains vanishing as the hypothesis space approaches full rank. These findings support budget allocation policies grounded in geometric criteria.

Abstract

Scientific discovery saturates when new hypotheses cease to provide independent information, even if the nominal hypothesis space remains large. We study hybrid discovery systems that combine structured local search with LLM-generated non-local proposals and pose the Search Compression Hypothesis: non-local exploration helps only when three geometric conditions co-occur: spectral compression, orthogonal escape from the explored span, and residual signal alignment with the target. We formalize these conditions, derive necessary conditions for hybrid advantage, and test the mechanism in controlled synthetic environments, large-scale A-share factor discovery, and symbolic-regression benchmarks; a public tabular operational sanity check tests the associated budget-allocation implication. Signal-planting and directed-versus-random experiments show that novelty alone is insufficient: random orthogonal jumps expand coverage but do not improve yield without predictive alignment. Across compression sweeps, real factor archives, and LLM-SRBench tasks, hybrid gains concentrate in weakly represented but target-bearing directions and vanish as the hypothesis space approaches full rank. The framework turns LLM-guided discovery from generic novelty search into a diagnostic procedure for deciding when directed non-local exploration is warranted.

Problem

• hypothesis redundancy
• scientific discovery
• discovery bottlenecks
• non-local exploration
• hybrid discovery systems

Innovation

• Search Compression Hypothesis
• hybrid discovery systems
• spectral compression
• orthogonal escape
• residual signal alignment