Reliable data clustering with Bayesian community detection

2025-10-16

AI Summary

Existing clustering methods—including k-means, hierarchical clustering, and WGCNA—lack principled model selection mechanisms and are highly sensitive to noise; while sparsification of similarity matrices mitigates noise, it relies on ad hoc thresholds that often distort true modular structure. This paper proposes a Bayesian community detection framework that jointly models similarity matrix sparsification and module identification. It integrates the degree-corrected stochastic block model (DCSBM), regularized Map Equation, and the Minimum Description Length (MDL) principle to enable fully automatic, threshold-free model selection. The method demonstrates robustness in high-dimensional, high-noise, and small-sample regimes, accurately recovering ground-truth modules. Experiments show that it significantly outperforms state-of-the-art methods on synthetic benchmarks and, when applied to gene co-expression networks, identifies gene modules with superior functional coherence and higher cross-dataset reproducibility.

Technology Category

• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Machine Learning: Clustering
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

From neuroscience and genomics to systems biology and ecology, researchers rely on clustering similarity data to uncover modular structure. Yet widely used clustering methods, such as hierarchical clustering, k-means, and WGCNA, lack principled model selection, leaving them susceptible to noise. A common workaround sparsifies a correlation matrix representation to remove noise before clustering, but this extra step introduces arbitrary thresholds that can distort the structure and lead to unreliable results. To detect reliable clusters, we capitalize on recent advances in network science to unite sparsification and clustering with principled model selection. We test two Bayesian community detection methods, the Degree-Corrected Stochastic Block Model and the Regularized Map Equation, both grounded in the Minimum Description Length principle for model selection. In synthetic data, they outperform traditional approaches, detecting planted clusters under high-noise conditions and with fewer samples. Compared to WGCNA on gene co-expression data, the Regularized Map Equation identifies more robust and functionally coherent gene modules. Our results establish Bayesian community detection as a principled and noise-resistant framework for uncovering modular structure in high-dimensional data across fields.

Problem

• Develops principled clustering methods resistant to noise in data
• Unifies sparsification and clustering with Bayesian model selection
• Detects reliable modular structures in high-dimensional scientific data

Innovation

• Unites sparsification and clustering via model selection
• Uses Bayesian community detection with Minimum Description Length
• Detects reliable clusters under high-noise conditions