From Guess2Graph: When and How Can Unreliable Experts Safely Boost Causal Discovery in Finite Samples?

2025-10-16

AI Summary

Causal discovery under few-shot settings suffers from severe performance degradation; existing approaches integrating expert knowledge—including large language models—rely on precise uncertainty quantification or perfect priors, exhibiting poor robustness. Method: We propose Guess2Graph, a novel framework that decouples expert “guesses” from statistical conditional independence (CI) testing: experts solely prioritize the order of CI tests without replacing the tests themselves. This ensures statistical consistency even when expert guidance is entirely erroneous, while gPC-Guess theoretically guarantees superior finite-sample performance over the standard PC algorithm whenever expert accuracy exceeds random chance. The method requires no expert uncertainty modeling and supports dynamic test sequencing. Contribution/Results: Experiments demonstrate monotonic improvement in structural recovery accuracy with increasing expert accuracy; under high-quality expert input, Guess2Graph significantly outperforms baselines, substantially enhancing causal structure learning in low-data regimes.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Causal discovery algorithms often perform poorly with limited samples. While integrating expert knowledge (including from LLMs) as constraints promises to improve performance, guarantees for existing methods require perfect predictions or uncertainty estimates, making them unreliable for practical use. We propose the Guess2Graph (G2G) framework, which uses expert guesses to guide the sequence of statistical tests rather than replacing them. This maintains statistical consistency while enabling performance improvements. We develop two instantiations of G2G: PC-Guess, which augments the PC algorithm, and gPC-Guess, a learning-augmented variant designed to better leverage high-quality expert input. Theoretically, both preserve correctness regardless of expert error, with gPC-Guess provably outperforming its non-augmented counterpart in finite samples when experts are "better than random." Empirically, both show monotonic improvement with expert accuracy, with gPC-Guess achieving significantly stronger gains.

Problem

• Improving causal discovery with limited sample data
• Safely integrating unreliable expert knowledge into algorithms
• Maintaining statistical consistency while boosting performance

Innovation

• Uses expert guesses to guide statistical tests
• Preserves correctness regardless of expert errors
• Learning-augmented variant leverages quality expert input