Score-informed Neural Operator for Enhancing Ordering-based Causal Discovery

2025-08-18

AI Summary

Existing score-based causal discovery methods face computational inefficiency, high memory consumption, and numerical instability when estimating the diagonal of the log-density Hessian. To address these bottlenecks, we propose Score-informed Neural Operator (SciNO), which directly models the score function in a smooth functional space—bypassing explicit second-order differentiation. SciNO integrates autoregressive priors with a diffusion modeling framework and employs neural operator networks to jointly approximate both the score function and its Hessian diagonal. Optimization leverages Stein gradient estimation and probabilistic control, enabling plug-and-play enhancement of causal reasoning in large language models without fine-tuning. Experiments demonstrate that SciNO reduces sequential divergence error by 42.7% on synthetic data and by 31.5% on real-world benchmarks compared to DiffAN, while maintaining memory efficiency and strong scalability.

Technology Category

• Machine Learning: Causal Learning
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Ordering-based approaches to causal discovery identify topological orders of causal graphs, providing scalable alternatives to combinatorial search methods. Under the Additive Noise Model (ANM) assumption, recent causal ordering methods based on score matching require an accurate estimation of the Hessian diagonal of the log-densities. However, previous approaches mainly use Stein gradient estimators, which are computationally expensive and memory-intensive. Although DiffAN addresses these limitations by substituting kernel-based estimates with diffusion models, it remains numerically unstable due to the second-order derivatives of score models. To alleviate these problems, we propose Score-informed Neural Operator (SciNO), a probabilistic generative model in smooth function spaces designed to stably approximate the Hessian diagonal and to preserve structural information during the score modeling. Empirical results show that SciNO reduces order divergence by 42.7% on synthetic graphs and by 31.5% on real-world datasets on average compared to DiffAN, while maintaining memory efficiency and scalability. Furthermore, we propose a probabilistic control algorithm for causal reasoning with autoregressive models that integrates SciNO's probability estimates with autoregressive model priors, enabling reliable data-driven causal ordering informed by semantic information. Consequently, the proposed method enhances causal reasoning abilities of LLMs without additional fine-tuning or prompt engineering.

Problem

• Stable Hessian diagonal approximation for causal ordering
• Memory-efficient score modeling in causal discovery
• Enhancing LLM causal reasoning without fine-tuning

Innovation

• Probabilistic generative model for Hessian diagonal approximation
• Memory-efficient scalable causal ordering with score modeling
• Probabilistic control algorithm integrating autoregressive priors