LISTA-Transformer Model Based on Sparse Coding and Attention Mechanism and Its Application in Fault Diagnosis

2026-03-04

AI Summary

This work addresses the limitations of existing deep models in simultaneously capturing local features and global dependencies, which often suffer from high computational complexity and poor interpretability. To overcome these challenges, the authors propose a novel sparse Transformer that integrates the Learnable Iterative Shrinkage-Thresholding Algorithm (LISTA) with Vision Transformer architecture, thereby embedding LISTA-based sparse coding deeply into the Transformer for the first time. This integration enables an adaptive mechanism for synergistic local–global feature representation. Vibration signals are transformed into time–frequency representations via continuous wavelet transform to enhance fault-related feature expression. Evaluated on the CWRU dataset, the proposed model achieves a fault recognition accuracy of 98.5%, outperforming conventional methods by 3.3% and surpassing existing Transformer-based approaches, while also improving both diagnostic accuracy and model interpretability.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Feature Construction/Reformulation
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Driven by the continuous development of models such as Multi-Layer Perceptron, Convolutional Neural Network (CNN), and Transformer, deep learning has made breakthrough progress in fields such as computer vision and natural language processing, and has been successfully applied in practical scenarios such as image classification and industrial fault diagnosis. However, existing models still have certain limitations in local feature modeling and global dependency capture. Specifically, CNN is limited by local receptive fields, while Transformer has shortcomings in effectively modeling local structures, and both face challenges of high model complexity and insufficient interpretability. In response to the above issues, we proposes the following innovative work: A sparse Transformer based on Learnable Iterative Shrinkage Threshold Algorithm (LISTA-Transformer) was designed, which deeply integrates LISTA sparse encoding with visual Transformer to construct a model architecture with adaptive local and global feature collaboration mechanism. This method utilizes continuous wavelet transform to convert vibration signals into time-frequency maps and inputs them into LISTA-Transformer for more effective feature extraction. On the CWRU dataset, the fault recognition rate of our method reached 98.5%, which is 3.3% higher than traditional methods and exhibits certain superiority over existing Transformer-based approaches.

Problem

• fault diagnosis
• local feature modeling
• global dependency capture
• model complexity
• interpretability

Innovation

• LISTA-Transformer
• sparse coding
• attention mechanism
• fault diagnosis
• time-frequency representation