A Three-Stage Bayesian Transfer Learning Framework to Improve Predictions in Data-Scarce Domains

2025-10-30

AI Summary

Deep learning models struggle with robust training under data scarcity due to sparse and noisy labels. Method: This paper proposes a three-stage Bayesian transfer learning framework: (1) initializing the feature extractor via pre-trained parameter transfer; (2) aligning source and target domain distributions in a shared latent space using Domain-Adversarial Neural Networks (DANN) to mitigate both domain shift and conditional shift; and (3) embedding a Bayesian neural network to yield calibrated uncertainty quantification. Contribution/Results: The framework seamlessly integrates deterministic representation learning, adversarial adaptation, and Bayesian inference, significantly improving prediction accuracy and generalization on the target domain. Evaluated on synthetic data and a real-world critical heat flux prediction task—transferring from tubular to rectangular channel geometries—the method outperforms state-of-the-art transfer approaches. It establishes an interpretable, robust modeling paradigm for small-sample, high-stakes domains such as nuclear engineering.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The use of ML in engineering has grown steadily to support a wide array of applications. Among these methods, deep neural networks have been widely adopted due to their performance and accessibility, but they require large, high-quality datasets. Experimental data are often sparse, noisy, or insufficient to build resilient data-driven models. Transfer learning, which leverages relevant data-abundant source domains to assist learning in data-scarce target domains, has shown efficacy. Parameter transfer, where pretrained weights are reused, is common but degrades under large domain shifts. Domain-adversarial neural networks (DANNs) help address this issue by learning domain-invariant representations, thereby improving transfer under greater domain shifts in a semi-supervised setting. However, DANNs can be unstable during training and lack a native means for uncertainty quantification. This study introduces a fully-supervised three-stage framework, the staged Bayesian domain-adversarial neural network (staged B-DANN), that combines parameter transfer and shared latent space adaptation. In Stage 1, a deterministic feature extractor is trained on the source domain. This feature extractor is then adversarially refined using a DANN in Stage 2. In Stage 3, a Bayesian neural network is built on the adapted feature extractor for fine-tuning on the target domain to handle conditional shifts and yield calibrated uncertainty estimates. This staged B-DANN approach was first validated on a synthetic benchmark, where it was shown to significantly outperform standard transfer techniques. It was then applied to the task of predicting critical heat flux in rectangular channels, leveraging data from tube experiments as the source domain. The results of this study show that the staged B-DANN method can improve predictive accuracy and generalization, potentially assisting other domains in nuclear engineering.

Problem

• Improving predictions in data-scarce domains using transfer learning
• Addressing domain shift instability in neural network training
• Providing calibrated uncertainty estimates for engineering applications

Innovation

• Three-stage Bayesian transfer learning framework
• Adversarially refined feature extractor with DANN
• Bayesian neural network for uncertainty quantification