RP-CATE: Recurrent Perceptron-based Channel Attention Transformer Encoder for Industrial Hybrid Modeling

2025-12-22

AI Summary

Existing industrial hybrid modeling approaches suffer from two key limitations: (1) architectural rigidity, and (2) inability to explicitly capture intrinsic structural relationships in data—such as monotonicity and periodicity. To address these, this paper proposes a novel physics-informed, data-driven encoder framework. It introduces a Transformer variant that replaces self-attention with channel-wise attention; designs pseudo-image data (PID) and pseudo-sequence data (PSD) representations; and integrates a recurrent sliding window mechanism with a Recurrent Perceptron (RP) module to explicitly encode underlying structural patterns in industrial time series. Additionally, sliding-window-based feature engineering is incorporated to enhance model interpretability. Evaluated on chemical process modeling tasks, the proposed method achieves significantly higher prediction accuracy than baseline models—including Transformer, LSTM, and MLP—while simultaneously attaining low computational overhead and improved interpretability.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Nowadays, industrial hybrid modeling which integrates both mechanistic modeling and machine learning-based modeling techniques has attracted increasing interest from scholars due to its high accuracy, low computational cost, and satisfactory interpretability. Nevertheless, the existing industrial hybrid modeling methods still face two main limitations. First, current research has mainly focused on applying a single machine learning method to one specific task, failing to develop a comprehensive machine learning architecture suitable for modeling tasks, which limits their ability to effectively represent complex industrial scenarios. Second, industrial datasets often contain underlying associations (e.g., monotonicity or periodicity) that are not adequately exploited by current research, which can degrade model's predictive performance. To address these limitations, this paper proposes the Recurrent Perceptron-based Channel Attention Transformer Encoder (RP-CATE), with three distinctive characteristics: 1: We developed a novel architecture by replacing the self-attention mechanism with channel attention and incorporating our proposed Recurrent Perceptron (RP) Module into Transformer, achieving enhanced effectiveness for industrial modeling tasks compared to the original Transformer. 2: We proposed a new data type called Pseudo-Image Data (PID) tailored for channel attention requirements and developed a cyclic sliding window method for generating PID. 3: We introduced the concept of Pseudo-Sequential Data (PSD) and a method for converting industrial datasets into PSD, which enables the RP Module to capture the underlying associations within industrial dataset more effectively. An experiment aimed at hybrid modeling in chemical engineering was conducted by using RP-CATE and the experimental results demonstrate that RP-CATE achieves the best performance compared to other baseline models.

Problem

• Develops a comprehensive machine learning architecture for industrial hybrid modeling tasks
• Exploits underlying associations in industrial datasets to improve predictive performance
• Enhances Transformer for industrial scenarios with channel attention and recurrent perceptron modules

Innovation

• Replaces self-attention with channel attention in Transformer
• Introduces Recurrent Perceptron module for capturing data associations
• Uses Pseudo-Image Data and cyclic sliding window for processing