Adaptive Dirichlet Process mixture model with unknown concentration parameter and variance: Scaling high dimensional clustering via collapsed variational inference

2026-01-28

AI Summary

Traditional Dirichlet process mixture models (DPMMs) struggle with adaptive clustering in high-dimensional data due to fixed hyperparameters and inefficient inference. This work proposes an adaptive DPMM based on collapsed variational inference that jointly models the concentration parameter α and the covariance structure for the first time, incorporating weakly informative priors to automatically infer both the number of clusters and their distributional shapes. The method substantially enhances clustering efficiency and robustness in high-dimensional settings, demonstrating faster convergence than state-of-the-art MCMC approaches in both Gaussian and negative binomial simulations. When applied to leukemia transcriptomic data, it not only accurately recovers known subtypes but also identifies novel, biologically meaningful subclusters.

Technology Category

• Machine Learning: Clustering
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

We propose a novel method that performs adaptive clustering with DPMM using collapsed VI, while incorporating weakly-informative priors for DP concentration parameter alpha and base distribution G0. We illustrate the importance of G0 covariance structure and prior choice by considering different parameterisations of the data covariance matrix. On high-dimensional Gaussian simulations, our model demonstrates substantially faster convergence than a state-of-the-art MCMC splice sampler. We further evaluate performances on Negative Binomial simulations and conduct sensitivity analyses to assess robustness on realistic data conditions. Application to a publicly available leukemia transcriptomic data set comprising 72 samples and 2,194 gene expression successfully recovers every known sub-type, all while identifying additional gene expression-based sub-clusters with meaningful biological interpretation.

Problem

• Dirichlet Process Mixture Model
• high-dimensional clustering
• concentration parameter
• variance estimation
• adaptive clustering

Innovation

• Dirichlet Process Mixture Model
• Collapsed Variational Inference
• Adaptive Clustering
• Weakly-informative Priors
• High-dimensional Covariance Structure