Foundation Models for Structural Health Monitoring

2024-04-03

IEEE Transactions on Sustainable Computing

AI Summary

To address the challenges of few-shot learning and high real-time requirements in structural health monitoring (SHM) for bridge structures—specifically anomaly detection (AD) and traffic load estimation (TLE)—this work pioneers the adaptation of a Masked Auto-Encoder–based Transformer architecture to SHM, establishing the first foundation-model paradigm for vibration time-series data. We propose a unified framework integrating self-supervised pretraining on multi-source vibration signals with downstream task fine-tuning, and design a knowledge distillation–based lightweighting strategy tailored for edge deployment. Experimental results demonstrate state-of-the-art performance: AD achieves 99.9% accuracy using only 15 time windows—substantially outperforming PCA; TLE attains an R² score of 0.97, representing a 6% improvement over prior art. Moreover, the model exhibits strong efficiency and compatibility with resource-constrained edge devices.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Structural Health Monitoring (SHM) is a critical task for ensuring the safety and reliability of civil infrastructures, typically realized on bridges and viaducts by means of vibration monitoring. In this paper, we propose for the first time the use of Transformer neural networks, with a Masked Auto-Encoder architecture, as Foundation Models for SHM. We demonstrate the ability of these models to learn generalizable representations from multiple large datasets through self-supervised pre-training, which, coupled with task-specific fine-tuning, allows them to outperform state-of-the-art traditional methods on diverse tasks, including Anomaly Detection (AD) and Traffic Load Estimation (TLE). We then extensively explore model size versus accuracy trade-offs and experiment with Knowledge Distillation (KD) to improve the performance of smaller Transformers, enabling their embedding directly into the SHM edge nodes. We showcase the effectiveness of our foundation models using data from three operational viaducts. For AD, we achieve a near-perfect 99.9% accuracy with a monitoring time span of just 15 windows. In contrast, a state-of-the-art method based on Principal Component Analysis (PCA) obtains its first good result (95.03% accuracy) only considering 120 windows. On two different TLE tasks, our models obtain state-of-the-art performance on multiple evaluation metrics (R$^2$ score, MAE% and MSE%). On the first benchmark, we achieve an R$^2$ score of 0.97 and 0.85 for light and heavy vehicle traffic, respectively, while the best previous approach stops at 0.91 and 0.84. On the second one, we achieve an R$^2$ score of 0.54 versus the 0.10 of the best existing method.

Problem

• Developing foundation models using Transformers for structural health monitoring
• Learning generalizable representations from multiple datasets via self-supervised pre-training
• Enabling accurate anomaly detection and traffic load estimation on bridges

Innovation

• Transformer neural networks with Masked Auto-Encoder architecture
• Self-supervised pre-training for generalizable representations from multiple datasets
• Knowledge Distillation to enhance smaller Transformers for edge nodes