Probabilistic data quality assessment for structural monitoring data via outlier-resistant conditional diffusion model

2026-04-29

AI Summary

This study addresses the challenge of data quality assessment in structural health monitoring under the influence of outliers by proposing a novel prediction-residual-based evaluation method. The approach employs a univariate implicit autoregressive conditional diffusion model, enhanced with a conditional embedding module to improve temporal modeling capacity. To increase robustness against distributional skewness and anomalous observations, it integrates interquartile normalization with Huber loss. The model outputs an anomaly probability for each data point and produces an overall quality score for the entire dataset. Experimental results on real-world structural monitoring data demonstrate that the proposed method significantly outperforms baseline techniques—including clustering, isolation-based anomaly detection, and deep reconstruction approaches—in terms of accuracy, robustness, and generalization capability.

Abstract

Data quality assessment is an essential step that ensures the reliability of the subsequent structural health monitoring (SHM) tasks. This study proposes a prediction deviation-based SHM data quality assessment method using a univariate implicit auto-regressive model, enabling outlier diagnosis and data cleaning. The proposed conditional diffusion model (CDM) augments the standard diffusion model with a conditional embedding module to incorporate temporal context, quartile normalization to mitigate distribution skew, and a Huber loss to enhance robustness against outliers. Within this univariate implicit autoregressive framework, each data point is assigned an outlier probability, quantifying its degree of "outlier-ness", and a global quality evaluation score is computed to characterize the overall dataset quality. Extensive case studies utilizing operational data from real-world structures demonstrate that the proposed framework significantly improves the accuracy of data quality assessment, outperforming other strong baselines representative of clustering, isolation-based, and deep reconstruction methods. The effectiveness and robustness of the proposed framework are further demonstrated by the findings of ablation experiments and hyperparameter analysis.

Problem

• data quality assessment
• structural health monitoring
• outlier detection
• probabilistic modeling
• time series data

Innovation

• conditional diffusion model
• outlier-resistant
• data quality assessment
• structural health monitoring
• Huber loss